Oculomotor learning revisited: a model of reinforcement learning in the basal ganglia incorporating an efference copy of motor actions
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
In its simplest formulation, reinforcement learning is based on the idea that if an action taken in a particular context is followed by a favorable outcome, then, in the same context, the tendency to produce that action should be strengthened, or reinforced. While reinforcement learning forms the basis of many current theories of basal ganglia (BG) function, these models do not incorporate distinct computational roles for signals that convey context, and those that convey what action an animal takes. Recent experiments in the songbird suggest that vocal-related BG circuitry receives two functionally distinct excitatory inputs. One input is from a cortical region that carries context information about the current "time" in the motor sequence. The other is an efference copy of motor commands from a separate cortical brain region that generates vocal variability during learning. Based on these findings, I propose here a general model of vertebrate BG function that combines context information with a distinct motor efference copy signal. The signals are integrated by a learning rule in which efference copy inputs gate the potentiation of context inputs (but not efference copy inputs) onto medium spiny neurons in response to a rewarded action. The hypothesis is described in terms of a circuit that implements the learning of visually guided saccades. The model makes testable predictions about the anatomical and functional properties of hypothesized context and efference copy inputs to the striatum from both thalamic and cortical sources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 2×
“, 2006), whose firing rate signals the appearance of unexpected rewards or the absence of rewards that were expected (Montague et al., 1996; Schultz et al., 1997; Schultz, 2002; Cohen et al., 2012).”
BackgroundLearning and Memory Functions of the Basal Ganglia
Packard, M. G., & Knowlton, B. J. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142937 · Cited by 1,717 · cited 3×
“Nor do striatal lesions prevent the animal from learning spatial aspects of the task, which are instead affected by hippocampal lesions (Packard et al., 1989; Packard and Knowlton, 2002; Featherstone and McDonald, 2004).”
BackgroundHabits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · cited 3×
“” The implementation of this principle, which forms the basis of reinforcement learning, instrumental learning, and stimulus-response learning (Sutton and Barto, 1998; Packard and Knowlton, 2002; Graybiel, 2008), requires three pieces of information: the action (response) that the animal makes, the context (situation)…”
BackgroundInferotemporal cortex and object vision
Tanaka, K. (1996) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.19.030196.000545 · Cited by 1,402
The Basal Ganglia and Adaptive Motor Control
Graybiel, A. M., Aosaki, T., Flaherty, A. W., & Kimura, M. (1994) · Science · doi:10.1126/science.8091209 · Cited by 1,324
Neuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318 · cited 2×
“, 2006), whose firing rate signals the appearance of unexpected rewards or the absence of rewards that were expected (Montague et al., 1996; Schultz et al., 1997; Schultz, 2002; Cohen et al., 2012).”
BackgroundThe Central Nervous System of Vertebrates
Nieuwenhuys, R., ten Donkelaar, H. J., & Nicholson, C. (1998) · doi:10.1007/978-3-642-18262-4 · Cited by 1,249
The basal ganglia: a vertebrate solution to the selection problem?
Redgrave, P., Prescott, T. J., & Gurney, K. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00319-4 · Cited by 1,199
Dichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190
Cited by
Differential Innervation of Direct- and Indirect-Pathway Striatal Projection Neurons
Wall, N. R., De La Parra, M., Callaway, E. M., & Kreitzer, A. C. (2013) · Neuron · doi:10.1016/j.neuron.2013.05.014 · Cited by 489
Birds, primates, and spoken language origins: behavioral phenotypes and neurobiological substrates
Petkov, C. I., & Jarvis, E. D. (2012) · Frontiers in Evolutionary Neuroscience · doi:10.3389/fnevo.2012.00012 · Cited by 409
Dopamine neurons encode performance error in singing birds
Gadagkar, V., Puzerey, P. A., Chen, R., Baird-Daniel, E., Farhang, A. R., & Goldberg, J. H. (2016) · Science · doi:10.1126/science.aah6837 · Cited by 268
The functional logic of corticostriatal connections
Shipp, S. (2017) · Brain Structure and Function · doi:10.1007/s00429-016-1250-9 · Cited by 118
“Yet the role-reversal model places the onus for action selection entirely with the broader origins of contextual input—sufficient for some circumstances, as illustrated by a schematic of a cue-driven oculomotor choice task (Fee 2012, 2014).”
BackgroundLearning Enhances Sensory Processing in Mouse V1 before Improving Behavior
Jurjut, O., Georgieva, P., Busse, L., & Katzner, S. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.3485-16.2017 · Cited by 99
Metadata
- Authors
- Fee, M. S.
- Venue
- Frontiers in Neural Circuits
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 38-38
- ISSN
- 1662-5110 (ISSN-L)
- OpenAlex
- W1971295726
- Cited by
- 40
- References
- 33
- FWCI
- 2.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.