The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Solving the Distal Reward Problem through Linkage of STDP and Dopamine Signaling
Abstract
In Pavlovian and instrumental conditioning, reward typically comes seconds after reward-triggering actions, creating an explanatory conundrum known as "distal reward problem": How does the brain know what firing patterns of what neurons are responsible for the reward if 1) the patterns are no longer there when the reward arrives and 2) all neurons and synapses are active during the waiting period to the reward? Here, we show how the conundrum is resolved by a model network of cortical spiking neurons with spike-timing-dependent plasticity (STDP) modulated by dopamine (DA). Although STDP is triggered by nearly coincident firing patterns on a millisecond timescale, slow kinetics of subsequent synaptic plasticity is sensitive to changes in the extracellular DA concentration during the critical period of a few seconds. Random firings during the waiting period to the reward do not affect STDP and hence make the network insensitive to the ongoing activity-the key feature that distinguishes our approach from previous theoretical studies, which implicitly assume that the network be quiet during the waiting period or that the patterns be preserved until the reward arrives. This study emphasizes the importance of precise firing patterns in brain dynamics and suggests how a global diffusive reinforcement signal in the form of extracellular DA can selectively influence the right synapses at the right time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Bi, G.-Q., & Poo, M.-M. (1998) · The Journal of Neuroscience · doi:10.1523/jneurosci.18-24-10464.1998 · Cited by 4,744
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564
Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs
Markram, H., Lübke, J., Frotscher, M., & Sakmann, B. (1997) · Science · doi:10.1126/science.275.5297.213 · Cited by 3,832
Competitive Hebbian learning through spike-timing-dependent synaptic plasticity
Song, S., Miller, K. D., & Abbott, L. F. (2000) · Nature Neuroscience · doi:10.1038/78829 · Cited by 2,729
Getting formal with dopamine and reward
Schultz, W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00967-4 · Cited by 2,574
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Synaptic tagging and long-term potentiation
Frey, U., & Morris, R. G. M. (1997) · Nature · doi:10.1038/385533a0 · Cited by 1,736
The principal features and mechanisms of dopamine modulation in the prefrontal cortex
Seamans, J. K., & Yang, C. R. (2004) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2004.05.006 · Cited by 1,592
Intrinsic firing patterns of diverse neocortical neurons
Connors, B. W., & Gutnick, M. J. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90185-d · Cited by 1,448
Cited by
Neurophysiological and Computational Principles of Cortical Rhythms in Cognition
Wang, X.-J. (2010) · Physiological Reviews · doi:10.1152/physrev.00035.2008 · Cited by 1,939
Neural syntax: cell assemblies, synapsembles, and readers
Buzsáki, G. (2010) · Neuron · doi:10.1016/j.neuron.2010.09.023 · Cited by 1,289
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126
The Spike-Timing Dependence of Plasticity
Feldman, D. E. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.001 · Cited by 1,071
GHOSH-DASTIDAR, S., & ADELI, H. (2009) · International Journal of Neural Systems · doi:10.1142/s0129065709002002 · Cited by 1,057
Large-scale model of mammalian thalamocortical systems
Izhikevich, E. M., & Edelman, G. M. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0712231105 · Cited by 1,021
Self-citationState-dependent computations: spatiotemporal processing in cortical networks
Buonomano, D. V., & Maass, W. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2558 · Cited by 926
Tang, J., Yuan, F., Shen, X., Wang, Z., Rao, M., He, Y., Sun, Y., Li, X., Zhang, W., Li, Y., et al. (2019) · Advanced Materials · doi:10.1002/adma.201902761 · Cited by 834
Synaptic Plasticity Forms and Functions
Magee, J. C., & Grienberger, C. (2020) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-090919-022842 · Cited by 806
Behavioral time scale synaptic plasticity underlies CA1 place fields
Bittner, K. C., Milstein, A. D., Grienberger, C., Romani, S., & Magee, J. C. (2017) · Science · doi:10.1126/science.aan3846 · Cited by 707
Decision Making in Recurrent Neuronal Circuits
Wang, X.-J. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.034 · Cited by 679
Updating dopamine reward signals
Schultz, W. (2013) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.11.012 · Cited by 651
Spiking Neural Networks and Their Applications: A Review
Yamazaki, K., Vo-Ho, V.-K., Bulsara, D., & Le, N. (2022) · Brain Sciences · doi:10.3390/brainsci12070863 · Cited by 626
Phenomenological models of synaptic plasticity based on spike timing
Morrison, A., Diesmann, M., & Gerstner, W. (2008) · Biological Cybernetics · doi:10.1007/s00422-008-0233-1 · Cited by 593
Ponulak, F., & Kasiński, A. (2010) · Neural Computation · doi:10.1162/neco.2009.11-08-901 · Cited by 573
SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks
Zenke, F., & Ganguli, S. (2018) · Neural Computation · doi:10.1162/neco_a_01086 · Cited by 515
Ghosh-Dastidar, S., & Adeli, H. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.04.003 · Cited by 490
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules
Frémaux, N., & Gerstner, W. (2016) · Frontiers in Neural Circuits · doi:10.3389/fncir.2015.00085 · Cited by 413
Zenke, F., Agnes, E. J., & Gerstner, W. (2015) · Nature Communications · doi:10.1038/ncomms7922 · Cited by 370
Introduction to spiking neural networks: Information processing, learning and applications
Ponulak, 2., & Kasinski, A. (2011) · Acta Neurobiologiae Experimentalis · doi:10.55782/ane-2011-1862 · Cited by 368
Map-based models in neuronal dynamics
Ibarz, B., Casado, J. M., & Sanjuán, M. A. F. (2011) · Physics Reports · doi:10.1016/j.physrep.2010.12.003 · Cited by 328
Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
Payeur, A., Guerguiev, J., Zenke, F., Richards, B. A., & Naud, R. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00857-x · Cited by 292
Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D., & Brea, J. (2018) · Frontiers in Neural Circuits · doi:10.3389/fncir.2018.00053 · Cited by 290
Song, H. F., Yang, G. R., & Wang, X.-J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004792 · Cited by 281
Computing with Spiking Neuron Networks
Paugam-Moisy, H., & Bohte, S. (2012) · Handbook of Natural Computing · doi:10.1007/978-3-540-92910-9_10 · Cited by 272
Legenstein, R., Pecevski, D., & Maass, W. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000180 · Cited by 267
What is reinforced by phasic dopamine signals?
Redgrave, P., Gurney, K., & Reynolds, J. (2008) · Brain Research Reviews · doi:10.1016/j.brainresrev.2007.10.007 · Cited by 261
Timing is not everything: neuromodulation opens the STDP gate
Pawlak, V. (2010) · Frontiers in Synaptic Neuroscience · doi:10.3389/fnsyn.2010.00146 · Cited by 255
Theoretical Neuroscience Rising
Abbott, L. F. (2008) · Neuron · doi:10.1016/j.neuron.2008.10.019 · Cited by 163
Single-cell transcriptomic evidence for dense intracortical neuropeptide networks
Smith, S. J., Sümbül, U., Graybuck, L. T., Collman, F., Seshamani, S., Gala, R., Gliko, O., Elabbady, L., Miller, J. A., Bakken, T. E., et al. (2019) · eLife · doi:10.7554/elife.47889 · Cited by 156
Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity
Coddington, L. T., & Dudman, J. T. (2019) · Neuron · doi:10.1016/j.neuron.2019.08.036 · Cited by 150
Pleger, B., Blankenburg, F., Ruff, C. C., Driver, J., & Dolan, R. J. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1093-08.2008 · Cited by 120
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
Hierarchical Address Event Routing for Reconfigurable Large-Scale Neuromorphic Systems
Park, J., Yu, T., Joshi, S., Maier, C., & Cauwenberghs, G. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2572164 · Cited by 118
The functional logic of corticostriatal connections
Shipp, S. (2017) · Brain Structure and Function · doi:10.1007/s00429-016-1250-9 · Cited by 118
Neurodynamics of mental exploration
Hopfield, J. J. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0913991107 · Cited by 96
Somato-dendritic synaptic plasticity and error-backpropagation in active dendrites
Schiess, M., Urbanczik, R., & Senn, W. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004638 · Cited by 75
Learning to Produce Syllabic Speech Sounds via Reward-Modulated Neural Plasticity
Warlaumont, A. S., & Finnegan, M. K. (2016) · PLOS ONE · doi:10.1371/journal.pone.0145096 · Cited by 71
Trace conditioning in insects—keep the trace!
Dylla, K. V., Galili, D. S., Szyszka, P., & Lüdke, A. (2013) · Frontiers in Physiology · doi:10.3389/fphys.2013.00067 · Cited by 57
Neural control of dopamine neurotransmission: implications for reinforcement learning
Aggarwal, M., Hyland, B. I., & Wickens, J. R. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08055.x · Cited by 32
Memory-Efficient Synaptic Connectivity for Spike-Timing- Dependent Plasticity
Pedroni, B. U., Joshi, S., Deiss, S. R., Sheik, S., Detorakis, G., Paul, S., Augustine, C., Neftci, E. O., & Cauwenberghs, G. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00357 · Cited by 29
Chorley, P., & Seth, A. K. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2011.00021 · Cited by 16
Reward magnitude determines reinforcement learning efficiency
Gong, S., Martell, A., Dudman, J. T., & Coddington, L. T. (2026) · Science · doi:10.1126/science.aeb0813 · Cited by 2
Metadata
- Authors
- Izhikevich, E. M.
- Venue
- Cerebral Cortex
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-01-13
- Volume
- 17
- Issue
- 10
- Pages
- 2443-2452
- ISSN
- 1047-3211 (ISSN-L)
- 1460-2199
- OpenAlex
- W4242770316
- Cited by
- 657
- References
- 46
- FWCI
- 13.30
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.