Transferring Learning from External to Internal Weights in Echo-State Networks with Sparse Connectivity
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Modifying weights within a recurrent network to improve performance on a task has proven to be difficult. Echo-state networks in which modification is restricted to the weights of connections onto network outputs provide an easier alternative, but at the expense of modifying the typically sparse architecture of the network by including feedback from the output back into the network. We derive methods for using the values of the output weights from a trained echo-state network to set recurrent weights within the network. The result of this "transfer of learning" is a recurrent network that performs the task without requiring the output feedback present in the original network. We also discuss a hybrid version in which online learning is applied to both output and recurrent weights. Both approaches provide efficient ways of training recurrent networks to perform complex tasks. Through an analysis of the conditions required to make transfer of learning work, we define the concept of a "self-sensing" network state, and we compare and contrast this with compressed sensing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025 · cited 1×
“To avoid these difficulties, Maass and collaborators [4] and Jaeger [5] suggested limiting synaptic modification during learning to the output weights, leaving the recurrent weights unchanged.”
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication
Jaeger, H., & Haas, H. (2004) · Science · doi:10.1126/science.1091277 · Cited by 3,811 · cited 4×
“The idea introduced by Jaeger and Haas [6], which allows learning to be restricted solely to the output weights w, is to change equation 1 for the network of Figure 1B to. t dx dt ~{xzJrzuzzvI(t)~{xz JzuwT rzvI(t), ð3Þ for the network of Figure 1A.”
BackgroundMethodInfluentialChaos in random neural networks
Sompolinsky, H., Crisanti, A., & Sommers, H. J. (1988) · Physical Review Letters · doi:10.1103/physrevlett.61.259 · Cited by 1,061 · cited 1×
“For most networks performing tasks that involve inputs and parameters with reasonable values, the PC eigenvalues fall rapidly, typically exponentially [15] [7] [16].”
Background
Cited by
Robust timing and motor patterns by taming chaos in recurrent neural networks
Laje, R., & Buonomano, D. V. (2013) · Nature Neuroscience · doi:10.1038/nn.3405 · Cited by 454
“…an RRN with recurrent plasticity (“fair recurrent plasticity”, which provided a control for the amount of plastic connections involved in the training); in this architecture the weights of the recurrent units are adjusted according to the error in the output unit (Sussillo and Abbott, 2009, 2012).”
BackgroundMethodPhysical reservoir computing—an introductory perspective
Nakajima, K. (2020) · Japanese Journal of Applied Physics · doi:10.35848/1347-4065/ab8d4f · Cited by 422
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules
Frémaux, N., & Gerstner, W. (2016) · Frontiers in Neural Circuits · doi:10.3389/fncir.2015.00085 · Cited by 413
A theory of multineuronal dimensionality, dynamics and measurement
Gao, P., Trautmann, E., Yu, B., Santhanam, G., Ryu, S., Shenoy, K., & Ganguli, S. (2017) · doi:10.1101/214262 · Cited by 332
Neural circuits as computational dynamical systems
Sussillo, D. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2014.01.008 · Cited by 247
“training just W, known as ‘reservoir computing’) [30,34–36], or to improved optimization algorithms [37,38 ,39–41], RNNs are enjoying a renaissance.”
Self-citation
Metadata
- Authors
- Sussillo, D., & Abbott, L. F.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2012-05-24
- Volume
- 7
- Issue
- 5
- Pages
- e37372
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2045416664
- Cited by
- 45
- References
- 3
- FWCI
- 1.90
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.