Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
SUMMARY Over a century of scientific work has focused on defining the factors motivating behavioral learning. Observations in animals and humans trained on a wide range of tasks support reinforcement learning (RL) algorithms as accounting for the learning. Still unknown, however, are the signals that drive learning in naïve, untrained subjects. Here, we capitalized on a sequential saccade task in which macaque monkeys acquired repetitive scanning sequences without instruction. We found that spike activity in the caudate nucleus after each trial corresponded to an integrated cost-benefit signal that was highly correlated with the degree of naturalistic untutored learning by the monkeys. Across learning, neurons encoding both cost and outcome gradually acquired increasingly sharp phasic trial-end responses that paralleled the development of the habit-like, repetitive saccade sequences. Our findings demonstrate a novel integrated cost-benefit signal by which RL and its neural correlates could drive naturalistic behaviors in freely behaving primates.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Marr, D. (1969) · The Journal of Physiology · doi:10.1113/jphysiol.1969.sp008820 · Cited by 3,253
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Computational principles of movement neuroscience
Wolpert, D. M., & Ghahramani, Z. (2000) · Nature Neuroscience · doi:10.1038/81497 · Cited by 2,122
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 4×
“…the specific coding of the difference between the actual and expected reward, reward prediction error (RPE), has been found in the striatum of humans (Daw et al., 2006; O'Doherty et al., 2004; Tanaka et al., 2004), monkeys (Asaad and Eskandar, 2011), and rodents (Stalnaker et al., 2012).”
InfluentialHabits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · cited 1×
“Many mechanisms have been proposed to drive the formation of such sequential actions and habits (Dezfouli and Balleine, 2012; Graybiel, 2008; Smith and Graybiel, 2014), one of which is reinforcement learning (RL).”
Self-citationControl of mental activities by internal models in the cerebellum
Ito, M. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2332 · Cited by 1,356 · cited 2×
“A prediction error in cost is not commonly used in modeling an adaptive change in behavior, but it bears resemblance to variables in traditional models of sequential (Squire, 2004) and non-sequential (Gomi and Kawato, 1993; Ito, 2008; Kawato and Gomi, 1992; Marr, 1969; Wolpert and Ghahramani, 2000) motor skill learnin…”
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Complementary roles of basal ganglia and cerebellum in learning and motor control
Doya, K. (2000) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00153-7 · Cited by 964
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Building neural representations of habits.
Jog, M. S., Kubota, Y., Connolly, C. I., Hillegaart, V., & Graybiel, A. M. (1999) · Science · doi:10.1126/science.286.5445.1745 · Cited by 834
Self-citationParallel neural networks for learning sequential procedures
Hikosaka, O., Nakahara, H., Rand, M. K., Sakai, K., Lu, X., Nakamura, K., Miyachi, S., & Doya, K. (1999) · Trends in Neurosciences · doi:10.1016/s0166-2236(99)01439-3 · Cited by 799
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · cited 4×
“…the specific coding of the difference between the actual and expected reward, reward prediction error (RPE), has been found in the striatum of humans (Daw et al., 2006; O'Doherty et al., 2004; Tanaka et al., 2004), monkeys (Asaad and Eskandar, 2011), and rodents (Stalnaker et al., 2012).”
InfluentialDifferent time courses of learning-related activity in the prefrontal cortex and striatum.
Pasupathy, A., & Miller, E. K. (2005) · Nature · doi:10.1038/nature03287 · Cited by 712 · cited 1×
“…have examined the short-term acquisition of new stimulus-response associations (Asaad and Eskandar, 2011; Histed et al., 2009; Kawagoe et al., 1998; Pasupathy and Miller, 2005; Samejima et al., 2005; Watanabe and Hikosaka, 2005; Williams and Eskandar, 2006), but none have examined the composition…”
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678 · cited 2×
“RL models have been extensively applied to model the acquisition of both simple and complex behaviors (Barraclough et al., 2004; Chukoskie et al., 2013; Daw et al., 2005, 2006), including movement sequences (Amemori et al.”
Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories
Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z., & Graybiel, A. M. (2005) · Nature · doi:10.1038/nature04053 · Cited by 595 · cited 8×
“To quantify a property of the neural activity that might reflect neural efficiency in encoding learning-related variables, we measured the sharpness of the activity in the Targ-Off window (Barnes et al., 2005; Jog et al., 1999).”
InfluentialSelf-citationProlonged dopamine signalling in striatum signals proximity and value of distant rewards
Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E. M., & Graybiel, A. M. (2013) · Nature · doi:10.1038/nature12475 · Cited by 565 · cited 1×
“Howe et al. (2013) found, with fast-scan cyclic voltammetry, that dopamine release in the striatum of rats ramps up with the progress towards a distant goal in a maze.”
Self-citationWhy Don't We Move Faster? Parkinson's Disease, Movement Vigor, and Implicit Motivation
Mazzoni, P., Hristova, A., & Krakauer, J. W. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0264-07.2007 · Cited by 557
Cerebellar complex spikes encode both destinations and errors in arm movements
Kitazawa, S., Kimura, T., & Yin, P.-B. (1998) · Nature · doi:10.1038/33141 · Cited by 401
Differential Corticostriatal Plasticity during Fast and Slow Motor Skill Learning in Mice
Costa, R. M., Cohen, D., & Nicolelis, M. A. L. (2004) · Current Biology · doi:10.1016/j.cub.2004.06.053 · Cited by 373 · cited 1×
“In rodents, chronic, long-term recordings have been made in the striatum during the acquisition of instructed habitual tasks (Barnes et al., 2005; Costa et al., 2004; Jin et al., 2014; Jog et al., 1999; Kimchi and Laubach, 2009; Smith and Graybiel, 2013).”
A dual operator view of habitual behavior reflecting cortical and striatal dynamics
Smith, K. S., & Graybiel, A. M. (2013) · Neuron · doi:10.1016/j.neuron.2013.05.038 · Cited by 324 · cited 4×
“It has further been shown that these end signals become impervious to reward devaluation, indicating that they likely are part of the neural signature of acquired habits (Smith and Graybiel, 2013).”
InfluentialSelf-citationA Corticostriatal Path Targeting Striosomes Controls Decision-Making under Conflict
Friedman, A., Homma, D., Gibb, L. G., Amemori, K.-I., Rubin, S. J., Hood, A. S., Riad, M. H., & Graybiel, A. M. (2015) · Cell · doi:10.1016/j.cell.2015.04.049 · Cited by 309 · cited 2×
“, 2011 for review), and both theoretical and experimental work specifically relating the effort of actions to the striatum has focused on cost/benefit trade-offs in motor tasks, positing a role of dopamine in their representation (Niv, 2007), or in decision-making tasks in which cost and benefit are combined variables…”
Self-citationHabits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297 · cited 3×
“…et al., 2004; Chukoskie et al., 2013; Daw et al., 2005, 2006), including movement sequences (Amemori et al., 2011; Desrochers et al., 2010; Dezfouli and Balleine, 2012), but the nature of the feedback error signal used in the uninstructed development of sequential, habit-like behaviors…”
Asaad, W. F., & Eskandar, E. N. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3793-11.2011 · Cited by 105
Amemori, K.-I., Amemori, S., & Graybiel, A. M. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1731-14.2015 · Cited by 49
Self-citation
Cited by
Feingold, J., Gibson, D. J., DePasquale, B., & Graybiel, A. M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517629112 · Cited by 370
Self-citationThe basal ganglia control the detailed kinematics of learned motor skills
Dhawale, A. K., Wolff, S. B. E., Ko, R., & Ölveczky, B. P. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00889-3 · Cited by 194
An insula hierarchical network architecture for active interoceptive inference
Fermin, A. S. R., Friston, K., & Yamawaki, S. (2022) · Royal Society Open Science · doi:10.1098/rsos.220226 · Cited by 101
Tanaka, Y. H., Tanaka, Y. R., Kondo, M., Terada, S.-I., Kawaguchi, Y., & Matsuzaki, M. (2018) · Neuron · doi:10.1016/j.neuron.2018.08.016 · Cited by 77
Marshall, J. J., Xu, J., & Contractor, A. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1788-17.2018 · Cited by 30
“How cortical information is transmitted into the basal ganglia circuit has implications for how this neural circuit promotes effective and discourages ineffective behavioral sequences (Desrochers et al., 2015; Nakamura et al., 2017).”
Background
Metadata
- Authors
- Desrochers, T. M., Amemori, K.-I., & Graybiel, A. M.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2015-08
- Volume
- 87
- Issue
- 4
- Pages
- 853-868
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W1908127931
- Cited by
- 54
- References
- 26
- FWCI
- 4.06
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.