The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Reinforcement Learning Through Modulation of Spike-Timing-Dependent Synaptic Plasticity
Abstract
The persistent modification of synaptic efficacy as a function of the relative timing of pre- and postsynaptic spikes is a phenomenon known as spike-timing-dependent plasticity (STDP). Here we show that the modulation of STDP by a global reward signal leads to reinforcement learning. We first derive analytically learning rules involving reward-modulated spike-timing-dependent synaptic and intrinsic plasticity, by applying a reinforcement learning algorithm to the stochastic spike response model of spiking neurons. These rules have several features common to plasticity mechanisms experimentally found in the brain. We then demonstrate in simulations of networks of integrate-and-fire neurons the efficacy of two simple learning rules involving modulated STDP. One rule is a direct extension of the standard STDP model (modulated STDP), and the other one involves an eligibility trace stored at each synapse that keeps a decaying memory of the relationships between the recent pairs of pre- and postsynaptic spike pairs (modulated STDP with eligibility trace). This latter rule permits learning even if the reward signal is delayed. The proposed rules are able to solve the XOR problem with both rate coded and temporally coded input and to learn a target output firing-rate pattern. These learning rules are biologically plausible, may be used for training generic artificial spiking neural networks, regardless of the neural model used, and suggest the experimental investigation in animals of the existence of reward-modulated STDP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Bi, G.-Q., & Poo, M.-M. (1998) · The Journal of Neuroscience · doi:10.1523/jneurosci.18-24-10464.1998 · Cited by 4,744
Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs
Markram, H., Lübke, J., Frotscher, M., & Sakmann, B. (1997) · Science · doi:10.1126/science.275.5297.213 · Cited by 3,832
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Competitive Hebbian learning through spike-timing-dependent synaptic plasticity
Song, S., Miller, K. D., & Abbott, L. F. (2000) · Nature Neuroscience · doi:10.1038/78829 · Cited by 2,729
Getting formal with dopamine and reward
Schultz, W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00967-4 · Cited by 2,574
Homeostatic plasticity in the developing nervous system
Turrigiano, G. G., & Nelson, S. B. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1327 · Cited by 2,485
Synaptic plasticity: taming the beast
Abbott, L. F., & Nelson, S. B. (2000) · Nature Neuroscience · doi:10.1038/81453 · Cited by 2,263 · cited 1×
“…(without considering the homeostatic mechanism introduced in section 2.2), we use an exponential dependence of plasticity on the relative spike timings , and we consider that the effect of different spike pairs is additive, as in previous studies (Song et al., 2000; Abbott & Nelson, 2000).”
MethodInfluentialThe principal features and mechanisms of dopamine modulation in the prefrontal cortex
Seamans, J. K., & Yang, C. R. (2004) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2004.05.006 · Cited by 1,592 · cited 2×
“It is known that dopamine and acetylcholine modulate classical (firing-rate-dependent) long-term potentiation and depression of synapses (Seamans & Yang, 2004; Huang, Simpson, Kellendonk, & Kandel, 2004; Thiel, Friston, & Dolan, 2002).”
BackgroundGerstner, W. (2009) · Encyclopedia of Neuroscience · doi:10.1016/b978-008045046-9.01405-4 · Cited by 1,199
Numerical recipes in C (2nd ed.): the art of scientific computing
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1992) · Cambridge University Press eBooks · Cited by 1,156
Spike Timing-Dependent Plasticity of Neural Circuits
Dan, Y., & Poo, M.-M. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.007 · Cited by 1,040
The other side of the engram: experience-driven changes in neuronal intrinsic excitability
Zhang, W., & Linden, D. J. (2003) · Nature Reviews Neuroscience · doi:10.1038/nrn1248 · Cited by 862
Spike-timing-dependent synaptic modification induced by natural spike trains
Froemke, R. C., & Dan, Y. (2002) · Nature · doi:10.1038/416433a · Cited by 803
Hebbian learning and spiking neurons
Kempter, R., Gerstner, W., & van Hemmen, J. L. (1999) · Physical Review E · doi:10.1103/physreve.59.4498 · Cited by 664
Synaptic plasticity in a cerebellum-like structure depends on temporal order
Bell, C. C., Han, V. Z., Sugawara, Y., & Grant, K. (1997) · Nature · doi:10.1038/387278a0 · Cited by 620
Phenomenological models of synaptic plasticity based on spike timing
Morrison, A., Diesmann, M., & Gerstner, W. (2008) · Biological Cybernetics · doi:10.1007/s00422-008-0233-1 · Cited by 593
Long-Term Plasticity of Intrinsic Excitability: Learning Rules and Mechanisms
Daoudal, G., & Debanne, D. (2003) · Learning & Memory · doi:10.1101/lm.64103 · Cited by 543
Spiking Neuron Models: An Introduction
Wulfram Gerstner, B., & Kistler, W. M. (2002) · Cambridge University Press eBooks · Cited by 528
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Cited by 525
Egger, V., Feldmeyer, D., & Sakmann, B. (1999) · Nature Neuroscience · doi:10.1038/16026 · Cited by 407
Learning Input Correlations through Nonlinear Temporally Asymmetric Hebbian Plasticity
Gütig, R., Aharonov, R., Rotter, S., & Sompolinsky, H. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-09-03697.2003 · Cited by 401
Infinite-Horizon Policy-Gradient Estimation
Baxter, J., & Bartlett, P. L. (2001) · Journal of Artificial Intelligence Research · doi:10.1613/jair.806 · Cited by 343
Learning in spiking neural networks by reinforcement of stochastic synaptic transmission
Seung, H. S. (2003) · Neuron · doi:10.1016/s0896-6273(03)00761-x · Cited by 340 · cited 5×
“This problem was also approached using other reinforcement learning methods for generic spiking neural networks (Seung, 2003; Xie & Seung, 2004).”
BackgroundMethodInfluentialSimulation-based optimization of Markov reward processes
Marbach, P., & Tsitsiklis, J. N. (2001) · IEEE Transactions on Automatic Control · doi:10.1109/9.905687 · Cited by 306
Pattern-recognizing stochastic learning automata
Barto, A. G., & Anandan, P. (1985) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1985.6313371 · Cited by 305
Aizenman, C. D., & Linden, D. J. (2000) · Nature Neuroscience · doi:10.1038/72049 · Cited by 270
Optimal Spike-Timing-Dependent Plasticity for Precise Action Potential Firing in Supervised Learning
Pfister, J.-P., Toyoizumi, T., Barber, D., & Gerstner, W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.6.1318 · Cited by 245
Huang, Y.-Y., Simpson, E., Kellendonk, C., & Kandel, E. R. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0308280101 · Cited by 231 · cited 1×
“It is known that dopamine and acetylcholine modulate classical (firing rate dependent) long term potentiation and depression of synapses (Seamans and Yang, 2004; Huang et al., 2004; Thiel et al., 2002).”
BackgroundA more biologically plausible learning rule for neural networks.
Mazzoni, P., Andersen, R. A., & Jordan, M. I. (1991) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.88.10.4433 · Cited by 227
Spike-Timing-Dependent Hebbian Plasticity as Temporal Difference Learning
Rao, R. P. N., & Sejnowski, T. J. (2001) · Neural Computation · doi:10.1162/089976601750541787 · Cited by 226
What Can a Neuron Learn with Spike-Timing-Dependent Plasticity?
Legenstein, R., Naeger, C., & Maass, W. (2005) · Neural Computation · doi:10.1162/0899766054796888 · Cited by 207
Learning by statistical cooperation of self-interested neuron-like computing elements.
Barto, A. (1985) · PubMed · Cited by 182
Spike timing dependent synaptic plasticity in biological systems
Roberts, P. D., & Bell, C. C. (2002) · Biological Cybernetics · doi:10.1007/s00422-002-0361-y · Cited by 171
Intrinsic Stabilization of Output Rates by Spike-Based Hebbian Learning
Kempter, R., Gerstner, W., & Hemmen, J. L. V. (2001) · Neural Computation · doi:10.1162/089976601317098501 · Cited by 171
The evidence for neural information processing with precise spike-times: A survey
Bohte, S. M. (2004) · Natural Computing · doi:10.1023/b:naco.0000027755.02868.60 · Cited by 167
Long-Term Potentiation of Intrinsic Excitability in LV Visual Cortical Neurons
Cudmore, R. H., & Turrigiano, G. G. (2004) · Journal of Neurophysiology · doi:10.1152/jn.01059.2003 · Cited by 166
Learning in neural networks by reinforcement of irregular spiking
Xie, X. S., & Seung, H. S. (2004) · Physical Review E · doi:10.1103/physreve.69.041909 · Cited by 157
Experiments with Infinite-Horizon, Policy-Gradient Estimation
Baxter, J., Bartlett, P. L., & Weaver, L. (2001) · Journal of Artificial Intelligence Research · doi:10.1613/jair.807 · Cited by 156
Kosco, B. (1987) · Cited by 152
Reversible Associative Depression and Nonassociative Potentiation at a Parallel Fiber Synapse
Han, V. Z., Grant, K., & Bell, C. C. (2000) · Neuron · doi:10.1016/s0896-6273(00)00070-2 · Cited by 151
Enhancement of presynaptic neuronal excitability by correlated presynaptic and postsynaptic spiking
Ganguly, K., Kiss, L., & Poo, M. (2000) · Nature Neuroscience · doi:10.1038/79838 · Cited by 143
Cholinergic Modulation of Experience-Dependent Plasticity in Human Auditory Cortex
Thiel, C. M., Friston, K. J., & Dolan, R. J. (2002) · Neuron · doi:10.1016/s0896-6273(02)00801-2 · Cited by 143
Hebbian Depression of Isolated Neuromuscular Synapses in Vitro
Dan, Y., & Poo, M. (1992) · Science · doi:10.1126/science.1317971 · Cited by 127
Lin, Y., Min, M., Chiu, T., & Yang, H. (2003) · Journal of Neuroscience · doi:10.1523/jneurosci.23-10-04173.2003 · Cited by 126
Computational Consequences of Temporally Asymmetric Learning Rules: I. Differential Hebbian Learning
Roberts, P. D. (1999) · Journal of Computational Neuroscience · doi:10.1023/a:1008910918445 · Cited by 122
A Spiking Neural Network Model of an Actor-Critic Learning Agent
Potjans, W., Morrison, A., & Diesmann, M. (2008) · Neural Computation · doi:10.1162/neco.2008.08-07-593 · Cited by 111
Direct gradient-based reinforcement learning
Baxter, J., & Bartlett, P. L. (2002) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2000.856049 · Cited by 110
Li, C., Lu, J., Wu, C.-P., Duan, S., & Poo, M. (2004) · Neuron · doi:10.1016/s0896-6273(03)00847-x · Cited by 102
Reinforcement learning in populations of spiking neuronsPreprint
Urbanczik, R., & Senn, W. (2008) · Nature Precedings · doi:10.1038/nn.2264 · Cited by 99
Synaptic activity modulates presynaptic excitability
Nick, T. A., & Ribera, A. B. (2000) · Nature Neuroscience · doi:10.1038/72082 · Cited by 72
Spike-Timing-Dependent Plasticity and Relevant Mutual Information Maximization
Chechik, G. (2003) · Neural Computation · doi:10.1162/089976603321891774 · Cited by 65
Spike-based Learning Rules and Stabilization of Persistent Neural Activity
Xie, X. S., & Seung, H. S. (1999) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 64
Democratic reinforcement: A principle for brain function
Stassinopoulos, D., & Bak, P. (1995) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.51.5033 · Cited by 59
A reinforcement learning algorithm for spiking neural networks
Florian, R. V. (2005) · International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc.2005.13 · Cited by 55
Self-citationBohté, S. M., & Mozer, M. C. (2007) · Neural Computation · doi:10.1162/neco.2007.19.2.371 · Cited by 52
Approximate Gradient Methods in Policy-Space Optimization of Markov Reward Processes
Marbach, P., & Tsitsiklis, J. N. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1022145020786 · Cited by 50
Learning rules and network repair in spike-timing-based computation networks
Hopfield, J. J., & Brody, C. D. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2536316100 · Cited by 43
Versatility and adaptive performance
Alstrøm, P., & Stassinopoulos, D. (1995) · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · doi:10.1103/physreve.51.5027 · Cited by 31
A Gradient Learning Rule for the Tempotron
Urbanczik, R., & Senn, W. (2008) · Neural Computation · doi:10.1162/neco.2008.09-07-605 · Cited by 25
Spike-timing Dependent Plasticity and Mutual Information Maximization for a Spiking Neuron Model
Toyoizumi, T., Pfister, J.-P., Aihara, K., & Wulfram Gerstner, B. (2004) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 19
Hebbian Synaptic Modifications in Spiking Neurons that LearnPreprint
Bartlett, P. L., & Baxter, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07247 · Cited by 18
Reinforcement Learning in Continuous State and Action Space
Strösslin, T., & Gerstner, W. (2003) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 18
Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison
Kolodziejski, C., Porr, B., & Wörgötter, F. (2008) · Biological Cybernetics · doi:10.1007/s00422-007-0209-6 · Cited by 17
Homeostasis and Learning through Spike-Timing Dependent Plasticity
Abbott, L. F., & Wulfram Gerstner, B. (2003) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 16
Biologically realizable reward-modulated hebbian training for spiking neural networks
Ferrari, S., Mehta, B., Muro, G. D., VanDongen, A. M. J., & Henriquez, C. S. (2008) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4634039 · Cited by 12
Takita, K., & Hagiwara, M. (2005) · Systems and Computers in Japan · doi:10.1002/scj.10645 · Cited by 6
Phase Precession and Recession with STDP and Anti-STDP
Florian, R. V., & Muresan, R. C. (2006) · Lecture notes in computer science · doi:10.1007/11840817_75 · Cited by 4
Self-citation
Cited by
Loihi: A Neuromorphic Manycore Processor with On-Chip Learning
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. (2018) · IEEE Micro · doi:10.1109/mm.2018.112130359 · Cited by 3,897
Towards spike-based machine intelligence with neuromorphic computing
Roy, K., Jaiswal, A., & Panda, P. (2019) · Nature · doi:10.1038/s41586-019-1677-2 · Cited by 2,190
State-dependent computations: spatiotemporal processing in cortical networks
Buonomano, D. V., & Maass, W. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2558 · Cited by 926
Phenomenological models of synaptic plasticity based on spike timing
Morrison, A., Diesmann, M., & Gerstner, W. (2008) · Biological Cybernetics · doi:10.1007/s00422-008-0233-1 · Cited by 593
Ponulak, F., & Kasiński, A. (2010) · Neural Computation · doi:10.1162/neco.2009.11-08-901 · Cited by 573
Metadata
- Authors
- Florian, R. V.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2007-06
- Volume
- 19
- Issue
- 6
- Pages
- 1468-1502
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2126404188
- Cited by
- 358
- References
- 70
- FWCI
- 8.04
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.