Backpropagation through time: what it does and how to do it
Abstract
Basic backpropagation, which is a simple method now being widely used in areas like pattern recognition and fault diagnosis, is reviewed. The basic equations for backpropagation through time, and applications to areas like pattern recognition involving dynamic systems, systems identification, and control are discussed. Further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations, or true recurrent networks, and other practical issues arising with the method are described. Pseudocode is provided to clarify the algorithms. The chain rule for ordered derivatives-the theorem which underlies backpropagation-is briefly discussed. The focus is on designing a simpler version of backpropagation which can be translated into computer code and applied directly by neutral network users.>
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Backpropagation-through-time exploits the fact that a recurrent neural network can be regarded as a temporally unfolded feedforward neural network.
“A standard algorithm for training a recurrent neural network, backpropagation-through-time, exploits the fact that a recurrent neural network can be regarded as a temporally unfolded feedforward neural network (Werbos, 1990).”
From The Cerebro-Cerebellum as a Locus of Forward Model: A Review (2020) · cites this paper as Werbos, 1990
The biological implementations of backpropagation proposed above do not give a natural implementation of backpropagation through time for recurrent networks.
“The biological implementations of backpropagation proposed above, while applicable to feedforward networks, do not give a natural implementation of “backpropagation through time” (BPTT) (Werbos, 1990) for recurrent networks”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Werbos, 1990
Cites
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501 · see citing works
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686 · see citing works
Numerical Methods for Unconstrained Optimization and Nonlinear Equations
Dennis, J. E., & Schnabel, R. B. (1996) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611971200 · Cited by 7,673 · see citing works
Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,772 · see citing works
Self-citationBeyond regression : new fools for prediction and analysis in the behavioral sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,746 · see citing works
Self-citationGeneralization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 947 · see citing works
Self-citationLearning State Space Trajectories in Recurrent Neural Networks
Pearlmutter, B. A. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.263 · Cited by 659 · see citing works
Numerical Methods for Unconstrained Optimization and Nonlinear Equations.
Kass, R. E., Dennis, J. E., & Schnabel, R. B. (1985) · Journal of the American Statistical Association · doi:10.2307/2288097 · Cited by 610 · see citing works
Conjugate Gradient Methods with Inexact Searches
Shanno, D. F. (1978) · Mathematics of Operations Research · doi:10.1287/moor.3.3.244 · Cited by 485 · see citing works
The truck backer-upper: an example of self-learning in neural networks
Nguyen, & Widrow (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118723 · Cited by 449 · see citing works
A learning rule for asynchronous perceptrons with feedback in a combinatorial environment
Almeida, L. B. (1990) · IEEE Press eBooks · Cited by 280 · see citing works
Applications of advances in nonlinear sensitivity analysis
Werbos, P. J. (2005) · Springer eBooks · doi:10.1007/bfb0006203 · Cited by 272 · see citing works
Self-citationBackpropagation: past and future
Werbos (1988) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1988.23866 · Cited by 229 · see citing works
Consistency of HDP applied to a simple reinforcement learning problem
Werbos, P. J. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90088-3 · Cited by 195 · see citing works
Self-citationLearning state space trajectories in recurrent neural networks
Pearlmutter, B. A. (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118724 · Cited by 186 · see citing works
BACKPROPAGATION: PAST AND FUTURE
Werbos, P. J. (1988) · Cited by 171 · see citing works
Self-citationGeneric constraints on underspecified target trajectories
Jordan, M. I. (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118584 · Cited by 157 · see citing works
The Truck Backer-Upper: An Example of Self-Learning in Neural Networks
Nguyen, D., & Widrow, B. C. (1990) · Elsevier eBooks · doi:10.1016/b978-0-444-88400-8.50006-6 · Cited by 153 · see citing works
Adaptive Control Using Neural Networks
Narendra, K. S. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.003.0009 · Cited by 128 · see citing works
Maximizing long-term gas industry profits in two minutes in Lotus using neural network methods
Werbos, P. J. (1989) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.31036 · Cited by 55 · see citing works
Self-citationAdaptive State Representation and Estimation Using Recurrent Connectionist Networks
Williams, R. J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.003.0008 · Cited by 51 · see citing works
Comparing different neural network architectures for classifying handwritten digits
Guyon, Poujaud, Personnaz, et al. (1989) · doi:10.1109/ijcnn.1989.118570 · Cited by 41 · see citing works
Sawai, Waibei, Haffner, Miyatake, & Shikano (1989) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1989.118682 · Cited by 32 · see citing works
Recent Advances in Numerical Techniques for Large-Scale Optimization
Shanno, D. F. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.003.0011 · Cited by 28 · see citing works
GENERALIZATION OF BACKPROPAGATION TO RECURRENT AND HIGH-ORDER NETWORKS.
Pineda, F. J. (1987) · Cited by 21 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Reservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956 · see citing works
Knowledge Graph Embedding: A Survey of Approaches and Applications
Wang, Q., Mao, Z., Wang, B., & Guo, L. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2754499 · Cited by 2,654 · see citing works
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2918951 · Cited by 2,455 · see citing works
“The training of RNN Vol. 107, No. 8, August 2019 | PROCEEDINGS OF THE IEEE 1741 is based on backpropagation through time (BPTT) [29].”
Deep learning for smart manufacturing: Methods and applications
Wang, J., Ma, Y., Zhang, L., Gao, R. X., & Wu, D. (2018) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2018.01.003 · Cited by 1,740 · see citing works
Learning skillful medium-range global weather forecasting
Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., et al. (2023) · Science · doi:10.1126/science.adi2336 · Cited by 1,466 · see citing works
Explainable artificial intelligence: an analytical review
Angelov, P. P., Soares, E. A., Jiang, R., Arnold, N. I., & Atkinson, P. M. (2021) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1424 · Cited by 843 · see citing works
Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822 · see citing works
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821 · see citing works
Temporal data classification and forecasting using a memristor-based reservoir computing system
Moon, J., Ma, W., Shin, J. H., Cai, F., Du, C., Lee, S. H., & Lu, W. D. (2019) · Nature Electronics · doi:10.1038/s41928-019-0313-3 · Cited by 645 · see citing works
A Review of Machine Learning and IoT in Smart Transportation
Zantalis, F., Koulouras, G., Karabetsos, S., & Kandris, D. (2019) · Future Internet · doi:10.3390/fi11040094 · Cited by 625 · see citing works
Learning task-dependent distributed representations by backpropagation through structure
Goller, C., & Kuchler, A. (2002) · Proceedings of International Conference on Neural Networks (ICNN'96) · doi:10.1109/icnn.1996.548916 · Cited by 581 · see citing works
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning
Wan, Z., Li, H., He, H., & Prokhorov, D. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2879572 · Cited by 548 · see citing works
Jaeger, H. (2007) · Scholarpedia · doi:10.4249/scholarpedia.2330 · Cited by 459 · see citing works
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · see citing works
Atitallah, S. B., Driss, M., Boulila, W., & Ghézala, H. B. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100303 · Cited by 375 · see citing works
Deep learning in drug discovery: opportunities, challenges and future prospects
Lavecchia, A. (2019) · Drug Discovery Today · doi:10.1016/j.drudis.2019.07.006 · Cited by 353 · see citing works
An Intrusion Detection System Using a Deep Neural Network With Gated Recurrent Units
Xu, C., Shen, J., Du, X., & Zhang, F. (2018) · IEEE Access · doi:10.1109/access.2018.2867564 · Cited by 339 · see citing works
In situ training of feed-forward and recurrent convolutional memristor networks
Wang, Z., Li, C., Lin, P., Rao, M., Nie, Y., Song, W., Qiu, Q., Li, Y., Yan, P., Strachan, J. P., et al. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0089-1 · Cited by 307 · see citing works
Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification
Rodriguez, P., Cucurull, G., Gonzalez, J., Gonfaus, J. M., Nasrollahi, K., Moeslund, T. B., & Roca, F. X. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2662199 · Cited by 297 · see citing works
DiPietro, R., & Hager, G. D. (2020) · Handbook of Medical Image Computing and Computer Assisted Intervention · doi:10.1016/b978-0-12-816176-0.00026-0 · Cited by 293 · see citing works
Adversarial Geometry-Aware Human Motion Prediction
Gui, L.-Y., Wang, Y.-X., Liang, X., & Moura, J. M. F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_48 · Cited by 276 · see citing works
Neural circuits as computational dynamical systems
Sussillo, D. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2014.01.008 · Cited by 247 · see citing works
The role of population structure in computations through neural dynamics
Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F., & Ostojic, S. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01088-4 · Cited by 237 · see citing works
Learning to Track with Object Permanence
Tokmakov, P., Li, J., Burgard, W., & Gaidon, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01068 · Cited by 226 · see citing works
Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.11.007 · Cited by 202 · see citing works
Improving Multi-Step Prediction of Learned Time Series Models
Venkatraman, A., Hebert, M., & Bagnell, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9590 · Cited by 198 · see citing works
Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling
Richard, A., Kuehne, H., & Gall, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.140 · Cited by 195 · see citing works
Deep Recursive Network Embedding with Regular Equivalence
Tu, K., Cui, P., Wang, X., Yu, P. S., & Zhu, W. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220068 · Cited by 179 · see citing works
Machine learning for biochemical engineering: A review
Mowbray, M., Savage, T., Wu, C., Song, Z., Cho, B. A., Del Rio-Chanona, E. A., & Zhang, D. (2021) · Biochemical Engineering Journal · doi:10.1016/j.bej.2021.108054 · Cited by 188 · see citing works
Lillicrap, T. P., & Scott, S. H. (2013) · Neuron · doi:10.1016/j.neuron.2012.10.041 · Cited by 157 · see citing works
The Cerebro-Cerebellum as a Locus of Forward Model: A Review
Tanaka, H., Ishikawa, T., Lee, J., & Kakei, S. (2020) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2020.00019 · Cited by 155 · see citing works
OpenEI: An Open Framework for Edge Intelligence
Zhang, X., Wang, Y., Lu, S., Liu, L., xu, L., & Shi, W. (2019) · 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00182 · Cited by 139 · see citing works
“The values of the parameters are adjusted using the standard backpropagation method [23] during training.”
MethodSenseGen: A deep learning architecture for synthetic sensor data generation
Alzantot, M., Chakraborty, S., & Srivastava, M. (2017) · 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2017.7917555 · Cited by 153 · see citing works
Connectome-constrained networks predict neural activity across the fly visual system
Lappalainen, J. K., Tschopp, F. D., Prakhya, S., McGill, M., Nern, A., Shinomiya, K., Takemura, S.-Y., Gruntman, E., Macke, J. H., & Turaga, S. C. (2024) · Nature · doi:10.1038/s41586-024-07939-3 · Cited by 128 · see citing works
“…differentiable 61 DMN model of the fly visual system can be interpreted as a continuous-time neural ordinary differential equation 62 with a deep convolutional recurrent neural network 63 architecture that is trained to perform a computer vision task using backpropagation through time 64,65 .”
Recurrent neural network-based approach for early recognition of Alzheimer's disease in EEG
Petrosian, A. A., Prokhorov, D. V., Lajara-Nanson, W., & Schiffer, R. B. (2001) · Clinical Neurophysiology · doi:10.1016/s1388-2457(01)00579-x · Cited by 126 · see citing works
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119 · see citing works
Long short-term memory-based deep recurrent neural networks for target tracking
Gao, C., Yan, J., Zhou, S., Varshney, P. K., & Liu, H. (2019) · Information Sciences · doi:10.1016/j.ins.2019.06.039 · Cited by 118 · see citing works
Classifying vortex wakes using neural networks
Colvert, B., Alsalman, M., & Kanso, E. (2018) · Bioinspiration & Biomimetics · doi:10.1088/1748-3190/aaa787 · Cited by 95 · see citing works
Reddy, S., Raghu, D., Khapra, M. M., & Joshi, S. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1036 · Cited by 93 · see citing works
Molinaro, D. D., Kang, I., Camargo, J., Gombolay, M. C., & Young, A. J. (2022) · IEEE Transactions on Medical Robotics and Bionics · doi:10.1109/tmrb.2022.3144025 · Cited by 90 · see citing works
Rotational dynamics in motor cortex are consistent with a feedback controller
Kalidindi, H. T., Cross, K. P., Lillicrap, T. P., Omrani, M., Falotico, E., Sabes, P. N., & Scott, S. H. (2021) · eLife · doi:10.7554/elife.67256 · Cited by 83 · see citing works
Rech, P. (2024) · IEEE Transactions on Nuclear Science · doi:10.1109/tns.2024.3349956 · Cited by 75 · see citing works
Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
Liu, J., Li, Y., Song, S., Xing, J., Lan, C., & Zeng, W. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2799968 · Cited by 62 · see citing works
Character-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works
Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network
Nishida, N., & Nakayama, H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-29451-3_54 · Cited by 53 · see citing works
A Holistic Feature Selection Method for Enhanced Short-term Load Forecasting of Power System
Jiang, B., Liu, Y., Geng, H., Wang, Y., Zeng, H., & Ding, J. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3219499 · Cited by 46 · see citing works
EDLA-EFDS: A Novel Ensemble Deep Learning Approach For Electrical Fault Detection Systems
Elmasry, W., & Wadi, M. (2022) · Electric Power Systems Research · doi:10.1016/j.epsr.2022.107834 · Cited by 42 · see citing works
Lappalainen, J. K., Tschopp, F. D., Prakhya, S., McGill, M., Nern, A., Shinomiya, K., Takemura, S.-Y., Gruntman, E., Macke, J. H., & Turaga, S. C. (2023) · doi:10.1101/2023.03.11.532232 · Cited by 31 · see citing works
Deep Learning and Dempster-Shafer Theory Based Insider Threat Detection
Tian, Z., Shi, W., Tan, Z., Qiu, J., Sun, Y., Jiang, F., & Liu, Y. (2024) · Mobile Networks and Applications · doi:10.1007/s11036-020-01656-7 · Cited by 24 · see citing works
Bonci, A., Fredianelli, L., Kermenov, R., Longarini, L., Longhi, S., Pompei, G., Prist, M., & Verdini, C. (2024) · Applied Sciences · doi:10.3390/app14198686 · Cited by 16 · see citing works
A Wide Area Measurement Based Neurocontrol for Generation Excitation Systems
Ray, S., & Venayagamoorthy, G. K. (2009) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2008.10.019 · Cited by 13 · see citing works
Quantized Reservoir Computing on Edge Devices for Communication Applications
Liu, S., Liu, L., & Yi, Y. (2020) · 2020 IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec50012.2020.00068 · Cited by 5 · see citing works
- Authors
- Werbos, P. J.
- Affiliations
- U.S. National Science Foundation (US)
- Corresponding author
- Werbos, P.J.
- ORCID
- Werbos, P.J.
- Venue
- Proceedings of the IEEE
- Published
- 1990
- Volume
- 78
- Issue
- 10
- Pages
- 1550-1560
- ISSN
- 0018-9219 (ISSN-L)
- OpenAlex
- W2150355110
- Cited by
- 4,879
- References
- 25
- Citations by year
- 2026: 161 · 2025: 244 · 2024: 309 · 2023: 352 · 2022: 298 · 2021: 392 · 2020: 380 · 2019: 366 · 2018: 335 · 2017: 227 · 2016: 220 · 2015: 113 · 2014: 86 · 2013: 61 · 2012: 69
- FWCI
- 25.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the IEEE (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
- Physical Sciences › Engineering › Control and Systems Engineering › Control Systems and Identification
- Keywords
- backpropagation through time, recurrent neural networks, dynamical systems, system identification, control, pattern recognition, fault diagnosis, systems of equations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
