Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Temporal models based on recurrent neural networks have proven to be quite powerful in a wide variety of applications, including language modeling and speech processing. However, training these models often relies on backpropagation through time (BPTT), which entails unfolding the network over many time steps, making the process of conducting credit assignment considerably more challenging. Furthermore, the nature of backpropagation itself does not permit the use of nondifferentiable activation functions and is inherently sequential, making parallelization of the underlying training process difficult. Here, we propose the parallel temporal neural coding network (P-TNCN), a biologically inspired model trained by the learning algorithm we call local representation alignment. It aims to resolve the difficulties and problems that plague recurrent networks trained by BPTT. The architecture requires neither unrolling in time nor the derivatives of its internal activation functions. We compare our model and learning procedure with other BPTT alternatives (which also tend to be computationally expensive), including real-time recurrent learning, echo state networks, and unbiased online recurrent optimization. We show that it outperforms these on-sequence modeling benchmarks such as Bouncing MNIST, a new benchmark we denote as Bouncing NotMNIST, and Penn Treebank. Notably, our approach can, in some instances, outperform full BPTT as well as variants such as sparse attentive backtracking. Significantly, the hidden unit correction phase of P-TNCN allows it to adapt to new data sets even if its synaptic weights are held fixed (zero-shot adaptation) and facilitates retention of prior generative knowledge when faced with a task sequence. We present results that show the P-TNCN's ability to conduct zero-shot adaptation and online continual sequence modeling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Neural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270 · cited 1×
“For every step of processing, the P-TNCN utilizes two types of recurrence/feedback loops to predict and correct states: 1) the model is recurrent across the temporal axis since it is stateful, which means that each processing layer depends on a vector summary of the past, and 2) the model is structurally recurrent, si…”
MethodGreff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · cited 1×
“For RTRL, SAB , BPTT, UORO, and the ESNs, we trained Long Short-Term Memory (LSTM) [60], [61] models with 1000 units and simple Elman recurrent network (RNN) models with 2000, tuning meta-parameters by tracking performance on the validation set.”
MethodFashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · cited 1×
“Other baseline learning procedures that we will compare to include BPTT [54], [55] and its truncated version, truncated BPTT (or TBPTT).”
MethodA Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369 · cited 1×
“Classically, a well-known online alternative to back-propagation of errors was real-time recurrent learning (RTRL, [23]), which employs forward-mode differentiation to compute gradients.”
BackgroundOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · cited 7×
“object bouncing around, which stands in contrast to what was discovered in [56] (which only investigated the case of Bouncing MNIST), where the LSTM trained with BPTT was found to “hallucinate” a second digit over top the first/original one.”
BackgroundMethodResultInfluentialProspective coding for objects in primate prefrontal cortex
Rainer, G., Rao, S. C., & Miller, E. K. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-13-05493.1999 · Cited by 432 · cited 2×
“The neuro-cognitive motivation behind the design of our proposed model and learning procedure is grounded in the principles of predictive coding and prospective coding [32].”
BackgroundMethodO'Reilly, R. C. (1996) · Neural Computation · doi:10.1162/neco.1996.8.5.895 · Cited by 352 · cited 1×
“Finally, as another means of introducing neuro-cognitively motivated regularization to our learned neural models we augment the update with a simple Hebbian component, motivated by early work done on LEABRA [49], [50] which did experiments showing that combining Hebbian learning with taskdriven (error-based) learning…”
MethodDynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex
Rao, R. P. N., & Ballard, D. H. (1997) · Neural Computation · doi:10.1162/neco.1997.9.4.721 · Cited by 290 · cited 3×
“Predictive coding, when applied to stateful problems, posits that many active processes in the brain, primarily those related to vision and speech, incrementally process stimuli sequences to build a dynamic, adaptable model of the world [18].”
BackgroundMethodA Neural Temporal Model for Human Motion Prediction
Gopalakrishnan, A., Mali, A., Kifer, D., Giles, L., & Ororbia, A. G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01239 · Cited by 185 · cited 1×
“These tasks range from those in statistical machine translation [6], to language modeling and text processing [7], [8], [9], to long and short-term human motion generation [10], to speech recognition [11].”
BackgroundSelf-citationIncremental backpropagation learning networks
Fu, L., Hsu, H.-H., & Principe, J. C. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.501732 · Cited by 103
Cited by
Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs
Millidge, B., Tschantz, A., & Buckley, C. L. (2022) · Neural Computation · doi:10.1162/neco_a_01497 · Cited by 95
Predictive coding networks for temporal prediction
Millidge, B., Tang, M., Osanlouy, M., Harper, N. S., & Bogacz, R. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011183 · Cited by 44
“In S3 Appendix we also examined a nonlinear task where the data generation process follows the generative model in Eqs 1 and 2 and obtained similar results.”
ResultMortal computation: a foundation for biomimetic intelligence
Ororbia, A., & Friston, K. (2023) · arXiv · doi:10.48550/arxiv.2311.09589 · Cited by 28
Self-citationA survey on neuro-mimetic deep learning via predictive coding
Salvatori, T., Mali, A., Buckley, C. L., Lukasiewicz, T., Rao, R. P. N., Friston, K., & Ororbia, A. (2026) · Neural Networks · doi:10.1016/j.neunet.2025.108161 · Cited by 2
Self-citation
Metadata
- Authors
- Ororbia, A., Mali, A., Giles, C. L., & Kifer, D.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2020-10
- Volume
- 31
- Issue
- 10
- Pages
- 4267-4278
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2904707853
- Cited by
- 57
- References
- 17
- FWCI
- 3.98
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.