A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem-action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Simple model of spiking neurons
Izhikevich, E. M. (2003) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2003.820440 · Cited by 4,828
Functional architecture of basal ganglia circuits: neural substrates of parallel processing
Alexander, G. E., & Crutcher, M. D. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90107-l · Cited by 4,131 · cited 3×
“For example, the somatotopic map found within the striatal motor territory is maintained throughout the basal ganglia circuit, such that there are separate channels for arm, leg, and face representations [18,98].”
BackgroundBienenstock, E., Cooper, L., & Munro, P. (1982) · The Journal of Neuroscience · doi:10.1523/jneurosci.02-01-00032.1982 · Cited by 2,896 · cited 1×
“This model showed how our plasticity rules have rate-coded (non-spiking) equivalents that are part of the well-known BCM family of learning rules [66,67].”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
From motivation to action: Functional interface between the limbic system and the motor system
MOGENSON, G., JONES, D., & YIM, C. (1980) · Progress in Neurobiology · doi:10.1016/0301-0082(80)90018-0 · Cited by 2,331 · cited 1×
“Since the tentative identification of the ventral striatum as this ‘‘limbicmotor’’ interface by Mogenson and colleagues [1], separate strands of work have elaborated four key elements centred on the striatum.”
BackgroundSynaptic plasticity: taming the beast
Abbott, L. F., & Nelson, S. B. (2000) · Nature Neuroscience · doi:10.1038/81453 · Cited by 2,263 · cited 1×
“The spike pair functions derived from the STDP, s+, were assigned a plausible time constant of 20 ms, based on similar, STDP counterparts [113].”
MethodA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“The error is derived from algorithms grounded in machine learning [63], but, in biological terms, it is often identified with phasic dopamine [2,64].”
MethodBasal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“The assumption of a channel architecture is based on the longstanding concept of parallel anatomical loops running throughout the basal ganglia nuclei [96,97].”
BackgroundRegulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry
Kravitz, A. V., Freeze, B. S., Parker, P. R. L., Kay, K., Thwin, M. T., Deisseroth, K., & Kreitzer, A. C. (2010) · Nature · doi:10.1038/nature09159 · Cited by 1,885
Context and Behavioral Processes in Extinction: Table 1.
Bouton, M. E. (2004) · Learning & Memory · doi:10.1101/lm.78804 · Cited by 1,808 · cited 2×
“To further test this hypothesis, we sought to determine whether the active suppression hypothesis could be reconciled with the post-extinction behavioural phenomenon of renewal (context-switch evoking immediate display of the previously acquired behaviour) and reacquisition of the key action (after a subsequent bout o…”
BackgroundHaber, S. N., Fudge, J. L., & McFarland, N. R. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-06-02369.2000 · Cited by 1,610
Two types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304
Phasic Firing in Dopaminergic Neurons Is Sufficient for Behavioral Conditioning
Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., & Deisseroth, K. (2009) · Science · doi:10.1126/science.1168878 · Cited by 1,266 · cited 1×
“They are consistent with the dominant hypothesis that phasic firing of dopamine neurons encodes a reward prediction error [2,3,5,37,38].”
ResultYin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217
Surmeier, D. J., Ding, J., Day, M., Wang, Z., & Shen, W. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.008 · Cited by 1,203 · cited 1×
“The network model included opposite effects of activating D1 and D2 receptors on MSN activity: D1 activation facilitated cortical efficacy at the input, while D2 activation attenuated this efficacy [45,104,105].”
BackgroundThe basal ganglia: a vertebrate solution to the selection problem?
Redgrave, P., Prescott, T. J., & Gurney, K. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00319-4 · Cited by 1,199
Self-citationDichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190
The role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099
Distinct roles for direct and indirect pathway striatal neurons in reinforcement
Kravitz, A. V., Tye, L. D., & Kreitzer, A. C. (2012) · Nature Neuroscience · doi:10.1038/nn.3100 · Cited by 972
A cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879
Dopamine-mediated regulation of corticostriatal synaptic plasticity
Calabresi, P., Picconi, B., Tozzi, A., & Di Filippo, M. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.001 · Cited by 865
Dynamic reorganization of striatal circuits during the acquisition and consolidation of a skill
Yin, H. H., Mulcare, S. P., Hilário, M. R. F., Clouse, E., Holloway, T., Davis, M. I., Hansson, A. C., Lovinger, D. M., & Costa, R. M. (2009) · Nature Neuroscience · doi:10.1038/nn.2261 · Cited by 818
The short-latency dopamine signal: a role in discovering novel actions?
Redgrave, P., & Gurney, K. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn2022 · Cited by 762
Self-citationDopamine signals for reward value and risk: basic and recent data
Schultz, W. (2010) · Behavioral and Brain Functions · doi:10.1186/1744-9081-6-24 · Cited by 653
Gremel, C. M., & Costa, R. M. (2013) · Nature Communications · doi:10.1038/ncomms3264 · Cited by 618 · cited 3×
“One candidate here is orbitofrontal cortex: it projects to the dorsomedial striatum [83], its neurons’ activity represents the expected value of an action [84,85], and optogenetic stimulation of its projection neurons promotes the maintenance of action during extinction [75] consistent with their encoding of value.”
BackgroundJoel, D., & Weiner, I. (2000) · Neuroscience · doi:10.1016/s0306-4522(99)00575-8 · Cited by 614 · cited 2×
“We consider the first two here, as basal ganglia circuitry is well-conserved between regions [80] (but see [81]) and it is not immediately clear how differences in the action selection mechanism could differentiate between outcome-driven and stimulus-driven behaviour.”
BackgroundActivity of striatal neurons reflects dynamic encoding and recoding of procedural memories
Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z., & Graybiel, A. M. (2005) · Nature · doi:10.1038/nature04053 · Cited by 595 · cited 1×
“Correspondingly, there is a re-organisation of single neuron activity in the dorsolateral striatum during habit formation [26,27,29,75].”
BackgroundA new perspective on the role of the orbitofrontal cortex in adaptive behaviour
Schoenbaum, G., Roesch, M. R., Stalnaker, T. A., & Takahashi, Y. K. (2009) · Nature reviews. Neuroscience · doi:10.1038/nrn2753 · Cited by 585
Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation
Faure, A., Haberland, U., Condé, F., & Massioui, N. E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3894-04.2005 · Cited by 402 · cited 2×
“Reflecting this, intact dopamine signalling in dorsolateral striatum is necessary for the formation of habitual behaviour [76], and blunting dopamine signalling prevents the formation of habitual behaviour but does not prevent goaldirected behaviour [89].”
BackgroundRedish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346 · cited 1×
“Thus in our model spontaneous post-extinction recovery arises solely from the plasticity rules without recourse to additional hypotheses such as state-space splitting proposed by the model of Redish and colleagues [94].”
MethodRenewal after the extinction of free operant behavior
Bouton, M. E., Todd, T. P., Vurbic, D., & Winterbauer, N. E. (2011) · Learning & Behavior · doi:10.3758/s13420-011-0018-6 · Cited by 281
Dopamine prediction error responses integrate subjective value from different reward dimensions
Lak, A., Stauffer, W. R., & Schultz, W. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1321596111 · Cited by 257 · cited 3×
“We proposed here a comprehensive modelling framework for testing this hypothesis, which links dopamine-modulated corticostriatal plasticity [6,8,15,17], phasic dopamine signals carrying environmental feedback [2,4,37,38], and the striatum’s role in reinforcement conditioned action selection [12–14].”
BackgroundResultSimilar roles of substantia nigra and ventral tegmental dopamine neurons in reward and aversion
Ilango, A., Kesner, A. J., Keller, K. L., Stuber, G. D., Bonci, A., & Ikemoto, S. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1703-13.2014 · Cited by 255 · cited 1×
“Particular challenges for this view are that dopamine signals to the striatum seem to encode the same information everywhere [90] (but see [91]) and the speed of change—if behavioural change depends solely on synaptic plasticity, then behaviour is likely altered slowly but the goal-directed system seems to rapidly ada…”
BackgroundContiguity and contingency in action-effect learning
Elsner, B., & Hommel, B. (2004) · Psychological Research · doi:10.1007/s00426-003-0151-8 · Cited by 243
Wörgötter, F., & Porr, B. (2005) · Neural Computation · doi:10.1162/0899766053011555 · Cited by 203
WICKENS, J. R., BUDD, C. S., HYLAND, B. I., & ARBUTHNOTT, G. W. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.016 · Cited by 165
Shan, Q., Ge, M., Christie, M. J., & Balleine, B. W. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.0313-14.2014 · Cited by 122 · cited 2×
“There is good evidence that synaptic weight change differs between the two regions in both skill-learning [92] and goal-directed learning [79], A New Framework for Cortico-Striatal Plasticity”
BackgroundChanges in activity of the striatum during formation of a motor habit
Tang, C., Pawlak, A. P., Prokopenko, V., & West, M. O. (2007) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2007.05353.x · Cited by 122
Todd, T. P., Winterbauer, N. E., & Bouton, M. E. (2012) · Appetite · doi:10.1016/j.appet.2011.12.006 · Cited by 60 · cited 3×
“To summarise the experimental data on ABA and ABB reacquisition performance from [50] we performed a linear regression on the data in their Figure 2 for each reacquisition context A or B (which was originally reported for ten time-points); the plotted symbols in Figure 8B are the endpoints of those regressions at the…”
BackgroundMethodInfluential
Cited by
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
What insects can tell us about the origins of consciousness.
Barron, A. B., & Klein, C. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1520084113 · Cited by 446
Early human motor development: From variation to the ability to vary and adapt
Hadders-Algra, M. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.05.009 · Cited by 338
Striatal Circuits as a Common Node for Autism Pathophysiology
Fuccillo, M. V. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00027 · Cited by 269
Cell-type-specific asynchronous modulation of PKA by dopamine in learning
Lee, S. J., Lodder, B., Chen, Y., Patriarchi, T., Tian, L., & Sabatini, B. L. (2021) · Nature · doi:10.1038/s41586-020-03050-5 · Cited by 160
Metadata
- Authors
- Gurney, K. N., Humphries, M. D., & Redgrave, P.
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-01-06
- Volume
- 13
- Issue
- 1
- Pages
- e1002034
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- OpenAlex
- W1976710123
- Cited by
- 129
- References
- 42
- FWCI
- 6.19
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.