Effect on movement selection of an evolving sensory representation: A multiple controller model of skill acquisition
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Change in behavior and neural activity in skill acquisition suggests that control is transferred from cortical planning areas (e.g., the prefrontal cortex, PFC) to the basal ganglia (BG). Planning has large computational and representational requirements but requires little experience with a task. The BG are thought to employ a simpler control scheme and reinforcement learning; these mechanisms rely on extensive experience. Many theoretical accounts of behavior in the face of uncertainty invoke planning mechanisms that explicitly take uncertainty into account. We suggest that the simpler mechanisms of the BG can also contribute to the development of behavior under such conditions. We focus on learning under conditions in which sensory information takes time to resolve, e.g., when a poorly perceived goal stimulus takes non-negligible time to identify. It may be advantageous to begin acting quickly under uncertainty — possibly via decisions that are suboptimal for the actual goal — rather than to wait for sensory information to fully resolve. We present a model of skill acquisition in which control is transferred, with experience, from a planning controller (denoted A), corresponding to the PFC, to a simpler controller (B), corresponding to the BG. We apply our model to a task in which a learning agent must execute a series of actions to achieve a goal (selected randomly at each trial from a small set). Over the course of a trial, the agent's goal representation evolves from representing all possible goals to only the selected goal. A is restricted to select movements only when goal representation is fully resolved. Model behavior is similar to that observed in humans accomplishing similar tasks. Thus, B can by itself account for the development of behavior under an evolving sensory representation, suggesting that the BG can contribute to learning and control under conditions of uncertainty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Formal Basis for the Heuristic Determination of Minimum Cost Paths
Hart, P., Nilsson, N., & Raphael, B. (1968) · IEEE Transactions on Systems Science and Cybernetics · doi:10.1109/tssc.1968.300136 · Cited by 12,738
Bayesian integration in sensorimotor learning
Körding, K. P., & Wolpert, D. M. (2004) · Nature · doi:10.1038/nature02169 · Cited by 2,004
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
The Basal Ganglia and Chunking of Action Repertoires
Graybiel, A. M. (1998) · Neurobiology of Learning and Memory · doi:10.1006/nlme.1998.3843 · Cited by 1,042 · cited 1×
“…in which the task is repeatedly accomplished, there is evidence that control may be transferred from cortical areas such as the prefrontal cortex (PFC) to the basal ganglia (BG) (Doyon and Benali, 2005; Packard and Knowlton, 2002; Jog et al., 1999; Puttemans et al., 2005; Graybiel, 1998).”
BackgroundReorganization and plasticity in the adult brain during learning of motor skills
Doyon, J., & Benali, H. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.03.004 · Cited by 1,028
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Interactions between frontal cortex and basal ganglia in working memory: A computational model
FRANK, M. J., LOUGHRY, B., & O'REILLY, R. C. (2001) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.1.2.137 · Cited by 879
Building neural representations of habits.
Jog, M. S., Kubota, Y., Connolly, C. I., Hillegaart, V., & Graybiel, A. M. (1999) · Science · doi:10.1126/science.286.5445.1745 · Cited by 834 · cited 1×
“…in which the task is repeatedly accomplished, there is evidence that control may be transferred from cortical areas such as the prefrontal cortex (PFC) to the basal ganglia (BG) (Doyon and Benali, 2005; Packard and Knowlton, 2002; Jog et al., 1999; Puttemans et al., 2005; Graybiel, 1998).”
BackgroundParallel neural networks for learning sequential procedures
Hikosaka, O., Nakahara, H., Rand, M. K., Sakai, K., Lu, X., Nakamura, K., Miyachi, S., & Doya, K. (1999) · Trends in Neurosciences · doi:10.1016/s0166-2236(99)01439-3 · Cited by 799
What's new with the amnesic patient H.M.?
Corkin, S. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn726 · Cited by 726
Different time courses of learning-related activity in the prefrontal cortex and striatum.
Pasupathy, A., & Miller, E. K. (2005) · Nature · doi:10.1038/nature03287 · Cited by 712 · cited 1×
“Though we do not explore it in this paper, there is evidence that cortical areas incorporate information provided by the basal ganglia (Pasupathy and Miller, 2005; Seger and Cincotta, 2006); computational models suggest how such information may aid cortex-based mechanisms (Houk and Wise, 1995;…”
BackgroundMidbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · cited 2×
“Briefly, the PFC is thought to employ planning mechanisms (Tanji and Hoshi, 2008; Mushiake et al., 2006; Miller and Cohen, 2001; Miller, 2000), while the BG are thought to employ a simpler scheme in which the expected value of each control choice — how " good " it is — is learned through experience…”
BackgroundNeural mechanisms of reward-related motor learning
Wickens, J. R., Reynolds, J. N., & Hyland, B. I. (2003) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2003.10.013 · Cited by 284 · cited 2×
“Early in learning, A dominated control: it selected the null action until b was fullyresolved and then selected actions that moved the agent directly to the true goal.”
BackgroundPuttemans, V., Wenderoth, N., & Swinnen, S. P. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3866-04.2005 · Cited by 280
Probabilistic models in human sensorimotor control
Wolpert, D. M. (2007) · Human Movement Science · doi:10.1016/j.humov.2007.05.005 · Cited by 212
Robust habit learning in the absence of awareness and independent of the medial temporal lobe
Bayley, P. J., Frascino, J. C., & Squire, L. R. (2005) · Nature · doi:10.1038/nature03857 · Cited by 176
Skill representation in the primary motor cortex after long-term practice
Matsuzaka, Y., Picard, N., & Strick, P. L. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00784.2006 · Cited by 156 · cited 1×
“In the realm of motor control, movements become faster and more coordinated (Matsuzaka et al., 2007; Hikosaka et al., 1995; Kent and Minifie, 1977; Abbs et al., 1984; Klein- Breteler et al., 2003; Engel et al., 1997; Baader et al., 2005; Jeannerod, 1981; Jerde et al., 2003) and come to be guided by…”
Background
Cited by
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346
“Due to this difference, the model of [64] can only be applied in cases where stimulus identification takes non-negligible time, which doesn’t seem to be the case of the experiments addressed by our model.”
MethodInfluentialHabits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297
“…devaluation on instrumental responses in different stages of learning (Daw et al. , 2005; Keramati et al. , 2011), the effect of habit formation on reaction times (Keramati et al. , 2011), and the effect of the complexity of state identification on the behavioral control (Shah & Barto, 2009).”
BackgroundEmbodied choice: how action influences perceptual decision making
Lepora, N. F., & Pezzulo, G. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004110 · Cited by 209
Evidence for model-based action planning in a sequential finger movement task
Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010) · Journal of Motor Behavior · doi:10.1080/00222895.2010.526467 · Cited by 36
“trol systems to guide behavior, although a few models have been proposed (Bapi & Doya, 2001; Daw, Niv, & Dayan, 2005; Shah and Barto, 2009).”
Background
Metadata
- Authors
- Shah, A., & Barto, A. G.
- Venue
- Brain Research
- Publisher
- Elsevier BV
- Published
- 2009-11
- Volume
- 1299
- Pages
- 55-73
- ISSN
- 0006-8993 (ISSN-L)
- OpenAlex
- W1977677629
- Cited by
- 16
- References
- 20
- FWCI
- 1.07
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.