Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Changes of synaptic connections between neurons are thought to be the physiological basis of learning. These changes can be gated by neuromodulators that encode the presence of reward. We study a family of reward-modulated synaptic learning rules for spiking neurons on a learning task in continuous space inspired by the Morris Water maze. The synaptic update rule modifies the release probability of synaptic transmission and depends on the timing of presynaptic spike arrival, postsynaptic action potentials, as well as the membrane potential of the postsynaptic neuron. The family of learning rules includes an optimal rule derived from policy gradient methods as well as reward modulated Hebbian learning. The synaptic update rule is implemented in a population of spiking neurons using a network architecture that combines feedforward input with lateral connections. Actions are represented by a population of hypothetical action cells with strong mexican-hat connectivity and are read out at theta frequency. We show that in this architecture, a standard policy gradient rule fails to solve the Morris watermaze task, whereas a variant with a Hebbian bias can learn the task within 20 trials, consistent with experiments. This result does not depend on implementation details such as the size of the neuronal populations. Our theoretical approach shows how learning new behaviors can be linked to reward-modulated plasticity at the level of single synapses and makes predictions about the voltage and spike-timing dependence of synaptic plasticity and the influence of neuromodulators such as dopamine. It is an important step towards connecting formal theories of reinforcement learning with neuronal and synaptic properties.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Malenka, R. C., & Bear, M. F. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.012 · Cited by 3,968
Bienenstock, E., Cooper, L., & Munro, P. (1982) · The Journal of Neuroscience · doi:10.1523/jneurosci.02-01-00032.1982 · Cited by 2,896
Simplified neuron model as a principal component analyzer
Oja, E. (1982) · Journal of Mathematical Biology · doi:10.1007/bf00275687 · Cited by 2,309 · cited 1×
“of the two neurons, the equation (1) captures the essence of a Hebb rule, i.e., the weight change depends only on the state of the two neurons, and possibly on the current value of the weight itself, but not on that of other neurons or other signals. Such a ‘2-factor’ Hebb rule is the basis of classical models of unsu…”
BackgroundSynaptic plasticity: taming the beast
Abbott, L. F., & Nelson, S. B. (2000) · Nature Neuroscience · doi:10.1038/81453 · Cited by 2,263 · cited 1×
“The first one consists in deriving a learning rule from reward optimization by gradient descent [32–34,38], an approach that can be linked to policy gradient methods in machine learning [39,40]; the second one postulates a form of STDP that is modulated by reward [33,35,36,41], an approach that can be considered an ex…”
BackgroundSynaptic tagging and long-term potentiation
Frey, U., & Morris, R. G. M. (1997) · Nature · doi:10.1038/385533a0 · Cited by 1,736 · cited 2×
“Such a picture has interesting relations to the model of synaptic tagging and capture [27] where synaptic connections undergo preliminary changes into early LTP or LTD that decay unless they are stabilized if plasticity related protein is available.”
BackgroundToward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 1×
“, or the early psychological theories of animal learning and conditioning by Sutton and Barto”
BackgroundProbabilistic decision making by slow reverberation in cortical circuits
Wang, X.-J. (2002) · Neuron · doi:10.1016/s0896-6273(02)01092-9 · Cited by 1,242
A cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879
The short-latency dopamine signal: a role in discovering novel actions?
Redgrave, P., & Gurney, K. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn2022 · Cited by 762
Cited by
Towards spike-based machine intelligence with neuromorphic computing
Roy, K., Jaiswal, A., & Panda, P. (2019) · Nature · doi:10.1038/s41586-019-1677-2 · Cited by 2,190
A Large-Scale Model of the Functioning Brain
Eliasmith, C., Stewart, T. C., Choo, X., Bekolay, T., DeWolf, T., Tang, Y., & Rasmussen, D. (2012) · Science · doi:10.1126/science.1225266 · Cited by 943
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules
Frémaux, N., & Gerstner, W. (2016) · Frontiers in Neural Circuits · doi:10.3389/fncir.2015.00085 · Cited by 413
Self-citationIntroduction to spiking neural networks: Information processing, learning and applications
Ponulak, 2., & Kasinski, A. (2011) · Acta Neurobiologiae Experimentalis · doi:10.55782/ane-2011-1862 · Cited by 368
Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018) · Frontiers in Neural Circuits · doi:10.3389/fncir.2018.00053 · Cited by 290
Self-citation
Metadata
- Authors
- Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., & Gerstner, W.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-12-04
- Volume
- 5
- Issue
- 12
- Pages
- e1000586
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2165061373
- Cited by
- 106
- References
- 14
- FWCI
- 2.74
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail. https://app.cochat.ai/papers/10.1371/journal.pcbi.1000586
- The paper
- Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., & Gerstner, W. (2009). Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1000586
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.