Oculomotor learning revisited: a model of reinforcement learning in the basal ganglia incorporating an efference copy of motor actions
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
In its simplest formulation, reinforcement learning is based on the idea that if an action taken in a particular context is followed by a favorable outcome, then, in the same context, the tendency to produce that action should be strengthened, or reinforced. While reinforcement learning forms the basis of many current theories of basal ganglia (BG) function, these models do not incorporate distinct computational roles for signals that convey context, and those that convey what action an animal takes. Recent experiments in the songbird suggest that vocal-related BG circuitry receives two functionally distinct excitatory inputs. One input is from a cortical region that carries context information about the current "time" in the motor sequence. The other is an efference copy of motor commands from a separate cortical brain region that generates vocal variability during learning. Based on these findings, I propose here a general model of vertebrate BG function that combines context information with a distinct motor efference copy signal. The signals are integrated by a learning rule in which efference copy inputs gate the potentiation of context inputs (but not efference copy inputs) onto medium spiny neurons in response to a rewarded action. The hypothesis is described in terms of a circuit that implements the learning of visually guided saccades. The model makes testable predictions about the anatomical and functional properties of hypothesized context and efference copy inputs to the striatum from both thalamic and cortical sources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 2×
“, 2006), whose firing rate signals the appearance of unexpected rewards or the absence of rewards that were expected (Montague et al., 1996; Schultz et al., 1997; Schultz, 2002; Cohen et al., 2012).”
BackgroundLearning and Memory Functions of the Basal Ganglia
Packard, M. G., & Knowlton, B. J. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142937 · Cited by 1,717 · cited 3×
“Nor do striatal lesions prevent the animal from learning spatial aspects of the task, which are instead affected by hippocampal lesions (Packard et al., 1989; Packard and Knowlton, 2002; Featherstone and McDonald, 2004).”
BackgroundHabits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · cited 3×
“” The implementation of this principle, which forms the basis of reinforcement learning, instrumental learning, and stimulus-response learning (Sutton and Barto, 1998; Packard and Knowlton, 2002; Graybiel, 2008), requires three pieces of information: the action (response) that the animal makes, the context (situation)…”
BackgroundInferotemporal cortex and object vision
Tanaka, K. (1996) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.19.030196.000545 · Cited by 1,402
The Basal Ganglia and Adaptive Motor Control
Graybiel, A. M., Aosaki, T., Flaherty, A. W., & Kimura, M. (1994) · Science · doi:10.1126/science.8091209 · Cited by 1,324
Neuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318 · cited 2×
“, 2006), whose firing rate signals the appearance of unexpected rewards or the absence of rewards that were expected (Montague et al., 1996; Schultz et al., 1997; Schultz, 2002; Cohen et al., 2012).”
BackgroundThe Central Nervous System of Vertebrates
Nieuwenhuys, R., ten Donkelaar, H. J., & Nicholson, C. (1998) · doi:10.1007/978-3-642-18262-4 · Cited by 1,249
The basal ganglia: a vertebrate solution to the selection problem?
Redgrave, P., Prescott, T. J., & Gurney, K. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00319-4 · Cited by 1,199
Dichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190
Revised nomenclature for avian telencephalon and some related brainstem nuclei
Reiner, A., Perkel, D. J., Bruce, L. L., Butler, A. B., Csillag, A., Kuenzel, W., Medina, L., Paxinos, G., Shimizu, T., Striedter, G., et al. (2004) · Journal of Comparative Neurology · doi:10.1002/cne.20118 · Cited by 1,148
The Basal Ganglia and Chunking of Action Repertoires
Graybiel, A. M. (1998) · Neurobiology of Learning and Memory · doi:10.1006/nlme.1998.3843 · Cited by 1,042 · cited 3×
“Neural circuitry in the basal ganglia (BG) is intimately involved in the control of learned behaviors (Graybiel et al., 1994; Graybiel, 1998), and is thought to be essential for the modification of behavior through reinforcement (Barto, 1995; Doya, 2000; Daw and Doya, 2006).”
BackgroundComplementary roles of basal ganglia and cerebellum in learning and motor control
Doya, K. (2000) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00153-7 · Cited by 964
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · cited 3×
“…signals allows the BG to determine which patterns of cortical activity are associated with, or predictive of, a favorable outcome and thus to bias motor and cognitive circuits toward favorable actions (Lauwereyns et al., 2002; Watanabe et al., 2003; Samejima et al., 2005; Frank and O’Reilly, 2006).”
BackgroundVisual-Motor Function of the Primate Superior Colliculus
Wurtz, R. H., & Albano, J. E. (1980) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.03.030180.001201 · Cited by 904 · cited 2×
“It appears, therefore, that such trial-and-error saccades are not initiated in the striatum, but are likely generated by one of the many of brain circuits that project to the superior colliculus, and are capable of triggering or influencing saccade generation (Wurtz and Albano, 1980).”
BackgroundA cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879
Neuronal correlates of parametric working memory in the prefrontal cortex
Romo, R., Brody, C. D., Hernández, A., & Lemus, L. (1999) · Nature · doi:10.1038/20939 · Cited by 822 · cited 1×
“…have been identified within the skeletomotor BG-thalamocortical circuit (Hoover and Strick, 1993), and there is some evidence for discrete topography in the output pathways of frontal eye fields (Robinson and Fuchs, 1969; Bruce and Goldberg, 1985; Komatsu and Suzuki, 1985; Schlag-Rey et al., 1992).”
BackgroundParallel neural networks for learning sequential procedures
Hikosaka, O., Nakahara, H., Rand, M. K., Sakai, K., Lu, X., Nakamura, K., Miyachi, S., & Doya, K. (1999) · Trends in Neurosciences · doi:10.1016/s0166-2236(99)01439-3 · Cited by 799
Making memories last: the synaptic tagging and capture hypothesis
Redondo, R. L., & Morris, R. G. M. (2011) · Nature Reviews Neuroscience · doi:10.1038/nrn2963 · Cited by 792 · cited 2×
“…a synaptic learning rule, perhaps related to gated Hebbian “triplet” learning rules (Farries and Fairhall, 2007; Fiete et al., 2007; Izhikevich, 2007; Redondo and Morris, 2011), that detects coincident activation of LMAN and HVC inputs, followed by a dopaminergic reward signal (Fee and Goldberg,…”
BackgroundGraybiel, A. M., & Ragsdale, C. W. (1978) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.75.11.5723 · Cited by 784
Different time courses of learning-related activity in the prefrontal cortex and striatum.
Pasupathy, A., & Miller, E. K. (2005) · Nature · doi:10.1038/nature03287 · Cited by 712 · cited 1×
“For example, monkeys can be trained to saccade toward or away from a particular stimulus (Kunimatsu and Tanaka, 2010), or can be trained to make saccades in a particular direction depending the identity of an object image, rather than its location (Pasupathy and Miller, 2005).”
BackgroundFreiwald, W. A., & Tsao, D. Y. (2010) · Science · doi:10.1126/science.1194908 · Cited by 678
A Pathway in Primate Brain for Internal Monitoring of Movements
Sommer, M. A., & Wurtz, R. H. (2002) · Science · doi:10.1126/science.1069590 · Cited by 565
Brain Circuits for the Internal Monitoring of Movements
Sommer, M. A., & Wurtz, R. H. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.060407.125627 · Cited by 508
Precise auditory–vocal mirroring in neurons for learned vocal communication
Prather, J. F., Peters, S., Nowicki, S., & Mooney, R. (2008) · Nature · doi:10.1038/nature06492 · Cited by 463 · cited 1×
“The HVC neurons projecting to Area X burst very sparsely, many generating a single highly reliable burst of spikes at one or a few specific moments in the song (Kozhevnikov and Fee, 2007; Prather et al., 2008; Fujimoto et al., 2011; Figure 2D).”
Background(1998) · doi:10.1016/s0165-0173(98)00016-2 · Cited by 398
Basal ganglia orient eyes to reward
Hikosaka, O., Nakamura, K., & Nakahara, H. (2006) · Journal of Neurophysiology · doi:10.1152/jn.00458.2005 · Cited by 377 · cited 8×
“Information about target stimuli in the oculomotor learning task is thought to be transmitted to the striatum by cortical inputs (Hikosaka et al., 2006).”
BackgroundInfluentialDendritic spines and distributed circuits
Yuste, R. (2011) · Neuron · doi:10.1016/j.neuron.2011.07.024 · Cited by 298 · cited 1×
“Such localization of synaptic signals and synaptic apparatus is thought to be one of the most important physiological functions of synaptic spines (Yuste, 2011).”
BackgroundCools, A. R. (1980) · Behavioural Brain Research · doi:10.1016/0166-4328(80)90035-2 · Cited by 283 · cited 2×
“These experiments support the idea that the cortical nucleus LMAN is the “variability generator” that drives vocal exploration during learning.”
BackgroundResultSeparate neural substrates for skill learning and performance in the ventral and dorsal striatum
Atallah, H. E., Lopez-Paniagua, D., Rudy, J. W., & O'Reilly, R. C. (2007) · Nature Neuroscience · doi:10.1038/nn1817 · Cited by 241 · cited 2×
“…an emerging view that the BG may not be the source of motor actions, but may serve to select from, or modify through learning, motor programs generated elsewhere (Cools, 1980; Mink, 1996; Redgrave et al., 1999; Gurney et al., 2001; Brown et al., 2004; Grillner et al., 2005; Atallah et al., 2007).”
BackgroundReward-Dependent Gain and Bias of Visual Responses in Primate Superior Colliculus
Ikeda, T., & Hikosaka, O. (2003) · Neuron · doi:10.1016/s0896-6273(03)00464-1 · Cited by 233 · cited 2×
“saccade direction was expressed by neurons in the SNr, which exhibited a suppression of activity only for rewarded saccades in a particular direction (Sato and Hikosaka, 2002), and in the superior colliculus, which exhibited increased activity in the rewarded direction (Ikeda and Hikosaka, 2003).”
BackgroundFiete, I. R., Fee, M. S., & Seung, H. S. (2007) · Journal of Neurophysiology · doi:10.1152/jn.01311.2006 · Cited by 167
Self-citationAronov, D., Veit, L., Goldberg, J. H., & Fee, M. S. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3009-11.2011 · Cited by 64 · cited 1×
“Furthermore, local brain cooling within LMAN results in slowing of the characteristic timescales of vocal babbling, suggesting that the biophysical and circuit dynamics within LMAN are involved in generating vocal variability (Aronov et al., 2011).”
Background
Cited by
Differential Innervation of Direct- and Indirect-Pathway Striatal Projection Neurons
Wall, N. R., De La Parra, M., Callaway, E. M., & Kreitzer, A. C. (2013) · Neuron · doi:10.1016/j.neuron.2013.05.014 · Cited by 489
Birds, primates, and spoken language origins: behavioral phenotypes and neurobiological substrates
Petkov, C. I., & Jarvis, E. D. (2012) · Frontiers in Evolutionary Neuroscience · doi:10.3389/fnevo.2012.00012 · Cited by 409
Dopamine neurons encode performance error in singing birds
Gadagkar, V., Puzerey, P. A., Chen, R., Baird-Daniel, E., Farhang, A. R., & Goldberg, J. H. (2016) · Science · doi:10.1126/science.aah6837 · Cited by 268
The functional logic of corticostriatal connections
Shipp, S. (2017) · Brain Structure and Function · doi:10.1007/s00429-016-1250-9 · Cited by 118
“Yet the role-reversal model places the onus for action selection entirely with the broader origins of contextual input—sufficient for some circumstances, as illustrated by a schematic of a cue-driven oculomotor choice task (Fee 2012, 2014).”
BackgroundLearning Enhances Sensory Processing in Mouse V1 before Improving Behavior
Jurjut, O., Georgieva, P., Busse, L., & Katzner, S. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.3485-16.2017 · Cited by 99
Metadata
- Authors
- Fee, M. S.
- Venue
- Frontiers in Neural Circuits
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 38-38
- ISSN
- 1662-5110 (ISSN-L)
- OpenAlex
- W1971295726
- Cited by
- 40
- References
- 33
- FWCI
- 2.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.