Effect on movement selection of an evolving sensory representation: A multiple controller model of skill acquisition
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Change in behavior and neural activity in skill acquisition suggests that control is transferred from cortical planning areas (e.g., the prefrontal cortex, PFC) to the basal ganglia (BG). Planning has large computational and representational requirements but requires little experience with a task. The BG are thought to employ a simpler control scheme and reinforcement learning; these mechanisms rely on extensive experience. Many theoretical accounts of behavior in the face of uncertainty invoke planning mechanisms that explicitly take uncertainty into account. We suggest that the simpler mechanisms of the BG can also contribute to the development of behavior under such conditions. We focus on learning under conditions in which sensory information takes time to resolve, e.g., when a poorly perceived goal stimulus takes non-negligible time to identify. It may be advantageous to begin acting quickly under uncertainty — possibly via decisions that are suboptimal for the actual goal — rather than to wait for sensory information to fully resolve. We present a model of skill acquisition in which control is transferred, with experience, from a planning controller (denoted A), corresponding to the PFC, to a simpler controller (B), corresponding to the BG. We apply our model to a task in which a learning agent must execute a series of actions to achieve a goal (selected randomly at each trial from a small set). Over the course of a trial, the agent's goal representation evolves from representing all possible goals to only the selected goal. A is restricted to select movements only when goal representation is fully resolved. Model behavior is similar to that observed in humans accomplishing similar tasks. Thus, B can by itself account for the development of behavior under an evolving sensory representation, suggesting that the BG can contribute to learning and control under conditions of uncertainty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Formal Basis for the Heuristic Determination of Minimum Cost Paths
Hart, P., Nilsson, N., & Raphael, B. (1968) · IEEE Transactions on Systems Science and Cybernetics · doi:10.1109/tssc.1968.300136 · Cited by 12,738
Bayesian integration in sensorimotor learning
Körding, K. P., & Wolpert, D. M. (2004) · Nature · doi:10.1038/nature02169 · Cited by 2,004
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
The Basal Ganglia and Chunking of Action Repertoires
Graybiel, A. M. (1998) · Neurobiology of Learning and Memory · doi:10.1006/nlme.1998.3843 · Cited by 1,042 · cited 1×
“…in which the task is repeatedly accomplished, there is evidence that control may be transferred from cortical areas such as the prefrontal cortex (PFC) to the basal ganglia (BG) (Doyon and Benali, 2005; Packard and Knowlton, 2002; Jog et al., 1999; Puttemans et al., 2005; Graybiel, 1998).”
BackgroundReorganization and plasticity in the adult brain during learning of motor skills
Doyon, J., & Benali, H. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.03.004 · Cited by 1,028
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Interactions between frontal cortex and basal ganglia in working memory: A computational model
FRANK, M. J., LOUGHRY, B., & O'REILLY, R. C. (2001) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.1.2.137 · Cited by 879
Cited by
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346
“Due to this difference, the model of [64] can only be applied in cases where stimulus identification takes non-negligible time, which doesn’t seem to be the case of the experiments addressed by our model.”
MethodInfluentialHabits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297
“…devaluation on instrumental responses in different stages of learning (Daw et al. , 2005; Keramati et al. , 2011), the effect of habit formation on reaction times (Keramati et al. , 2011), and the effect of the complexity of state identification on the behavioral control (Shah & Barto, 2009).”
BackgroundEmbodied choice: how action influences perceptual decision making
Lepora, N. F., & Pezzulo, G. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004110 · Cited by 209
Evidence for model-based action planning in a sequential finger movement task
Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010) · Journal of Motor Behavior · doi:10.1080/00222895.2010.526467 · Cited by 36
“trol systems to guide behavior, although a few models have been proposed (Bapi & Doya, 2001; Daw, Niv, & Dayan, 2005; Shah and Barto, 2009).”
Background
Metadata
- Authors
- Shah, A., & Barto, A. G.
- Venue
- Brain Research
- Publisher
- Elsevier BV
- Published
- 2009-11
- Volume
- 1299
- Pages
- 55-73
- ISSN
- 0006-8993 (ISSN-L)
- OpenAlex
- W1977677629
- Cited by
- 16
- References
- 20
- FWCI
- 1.07
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.