Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
The brain's most difficult computation in decision-making learning is searching for essential information related to rewards among vast multimodal inputs and then integrating it into beneficial behaviors. Contextual cues consisting of limbic, cognitive, visual, auditory, somatosensory, and motor signals need to be associated with both rewards and actions by utilizing an internal representation such as reward prediction and reward prediction error. Previous studies have suggested that a suitable brain structure for such integration is the neural circuitry associated with multiple cortico-striatal loops. However, computational exploration still remains into how the information in and around these multiple closed loops can be shared and transferred. Here, we propose a "heterarchical reinforcement learning" model, where reward prediction made by more limbic and cognitive loops is propagated to motor loops by spiral projections between the striatum and substantia nigra, assisted by cortical projections to the pedunculopontine tegmental nucleus, which sends excitatory input to the substantia nigra. The model makes several fMRI-testable predictions of brain activity during stimulus-action-reward association learning. The caudate nucleus and the cognitive cortical areas are correlated with reward prediction error, while the putamen and motor-related areas are correlated with stimulus-action-dependent reward prediction. Furthermore, a heterogeneous activity pattern within the striatum…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These included areas implicated in social cognition, such as the precuneus, medial prefrontal cortex, left temporal pole, lateral occipital cortex, hippocampus”
From The Power of the Like in Adolescence: Effects of Peer Influence on Neural and Behavioral Responses to Social Media (2016) · cites this paper as Haruno & Kawato, 2006
Citations
Cites
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 1×
“We examined these predictions by using model-based fMRI (Haruno et al., 2004; Haruno & Kawato, 2006; O’Doherty et al., 2004), where a computational model is utilized to estimate such internal subject variables as reward prediction and reward prediction error, which are then used in the correlation analysis of brain ac…”
MethodA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 2×
“However, in contrast to previous basal-ganglia reinforcement-learning models (Brown, Bullock, & Grossberg, 1999; Houk et al., 1995; Montague et al., 1996), we assume that PPTN excitatory inputs originating from the cortex provide the dorsomedial part of the VTA/SNc, the summation of time-advanced…”
ResultBasal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“However, it is well-known that each cortico-striatal loop is closed; a loop originating from a cortical region goes around the striatum, the globus pallidus, and the thalamus and then projects back to the original location without much blending (Middleton & Strick, 2000).”
BackgroundMotor Areas of the Medial Wall: A Review of Their Location and Functional Activation
Picard, N., & Strick, P. L. (1996) · Cerebral Cortex · doi:10.1093/cercor/6.3.342 · Cited by 1,712 · cited 2×
“These areas have been implicated in reward-related cognitive functions ( Picard & Strick, 1996; Price et al., 1996), and”
BackgroundHaber, S. N., Fudge, J. L., & McFarland, N. R. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-06-02369.2000 · Cited by 1,610
The primate basal ganglia: Parallel and integrative networks
Haber, S. N. (2003) · Journal of Chemical Neuroanatomy · doi:10.1016/j.jchemneu.2003.10.003 · Cited by 1,479 · cited 15×
“Furthermore, because of internal differences in the midbrain projections within the caudate nucleus and the putamen (Haber et al., 2000; Haber, 2003), correlation with the stimulus-action-dependent reward prediction in the putamen should be posterior in easy tasks but anterior in difficult tasks when we control learni…”
BackgroundMethodInfluentialNeuronal Coding of Prediction Errors
Schultz, W., & Dickinson, A. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.473 · Cited by 1,427 · cited 2×
“…with the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA), so it can therefore regulate multimodal inputs by the effect of dopamine (Hollerman & Schultz, 1998; Schultz, Apicella, Scarnati, & Ljungberg, 1992; Schultz & Dickinson, 2000; Takikawa, Kawagoe, & Hikosaka, 2004).”
BackgroundModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Central mechanisms of motor skill learning
Hikosaka, O., Nakamura, K., Sakai, K., & Nakahara, H. (2002) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(02)00307-0 · Cited by 1,023 · cited 1×
“Our model and experimental data are also consistent with a pioneering work from Hikosaka’s group that focused on the functional roles of parallel cortico-striatal loops during sequential motor learning of monkeys (Hikosaka et al., 1999; Hikosaka, Nakamura, Sakai, & Nakahara, 2002; Miyachi,Hikosaka,Miyashita,Karadi,RMi…”
ResultParallel neural networks for learning sequential procedures
Hikosaka, O., Nakahara, H., Rand, M. K., Sakai, K., Lu, X., Nakamura, K., Miyachi, S., & Doya, K. (1999) · Trends in Neurosciences · doi:10.1016/s0166-2236(99)01439-3 · Cited by 799
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678 · cited 1×
“Dynamic association of various contextual cues with action and reward is critical to make effective decisions ( Barraclough, Conroy, & Lee, 2004 ).”
BackgroundTwo distinct modes of sensory processing observed in monkey primary visual cortex (V1)
Supèr, H., Spekreijse, H., & Lamme, V. A. F. (2001) · Nature Neuroscience · doi:10.1038/85170 · Cited by 467
Differential roles of monkey striatum in learning of sequential hand movement
Miyachi, S., Hikosaka, O., Miyashita, K., Kárádi, Z., & Rand, M. K. (1997) · Experimental Brain Research · doi:10.1007/pl00005669 · Cited by 374 · cited 1×
“Our model and experimental data are also consistent with a pioneering work from Hikosaka’s group that focused on the functional roles of parallel cortico-striatal loops during sequential motor learning of monkeys (Hikosaka et al., 1999; Hikosaka, Nakamura, Sakai, & Nakahara, 2002; Miyachi,Hikosaka,Miyashita,Karadi,R M…”
ResultHaruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M., Samejima, K., Imamizu, H., & Kawato, M. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.3417-03.2004 · Cited by 300 · cited 2×
“We examined these predictions by using model-based fMRI (Haruno et al., 2004; Haruno & Kawato, 2006; O’Doherty et al., 2004), where a computational model is utilized to estimate such internal subject variables as reward prediction and reward prediction error, which are then used in the correlation analysis of brain ac…”
MethodSelf-citationMiyachi, S., Hikosaka, O., & Lu, X. (2002) · Experimental Brain Research · doi:10.1007/s00221-002-1213-7 · Cited by 262 · cited 1×
“Our model and experimental data are also consistent with a pioneering work from Hikosaka’s group that focused on the functional roles of parallel cortico-striatal loops during sequential motor learning of monkeys (Hikosaka et al., 1999; Hikosaka, Nakamura, Sakai, & Nakahara, 2002; Miyachi,Hikosaka,Miyashita,Karadi,R M…”
ResultKobayashi, Y., Inoue, Y., Yamamoto, M., Isa, T., & Aizawa, H. (2002) · Journal of Neurophysiology · doi:10.1152/jn.2002.88.2.715 · Cited by 104
Research works printed but not linked: 21
- Alexander, 1990, Basal ganglia thalamocortical circuits: Parallel substrates for motor, oculomotor, “prefrontal” and “limbic” functions, Progress in Brain Research, 85, 119, 10.1016/S0079-6123(08)62678-3 · DOI · Paper page
- Barto, 1983, Neuron-like elements that can solve difficult learning control problems, IEEE Transactions on Systems Man and Cybernetics, 13, 835
- Brown, 1999, How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues, Journal of Neuroscience, 19, 10502, 10.1523/JNEUROSCI.19-23-10502.1999 · DOI · Paper page
- Dayan, 1993, Feudal reinforcement learning, Vol. 5, 271
- Friston, 1995, Statistical parametric maps in functional brain imaging: A general linear approach, Human Brain Mapping, 2, 189, 10.1002/hbm.460020402 · DOI · Paper page
- Gerardin, 2003, Foot, hand, face and eye representation in the human striatum, Cerebral Cortex, 13, 162, 10.1093/cercor/13.2.162 · DOI · Paper page
- Haruno, 2001, Mosaic model for sensorimotor learning and control, Neural Computation, 13, 2201, 10.1162/089976601750541778 · DOI · Paper page
- Haruno, 2006, Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning, Journal of Neurophysiology, 92, 948, 10.1152/jn.00382.2005 · DOI · Paper page
- Hollerman, 1998, Dopamine neurons report an error in the temporal prediction of reward during learning, Nature Neuroscience, 4, 304, 10.1038/1124 · DOI · Paper page
- Morimoto, 1999, Hierarchical reinforcement learning for motion learning: learning “stand-up” trajectories, Advanced Robotics, 13, 267, 10.1163/156855399X01314 · doi:10.1163/156855399x01314
- Oakman, 1995, Distribution of pontomesencephalic cholinergic neurons projecting to substantia nigra differs significantly from those projecting to ventral tegmental area, Journal of Neuroscience, 15, 5859, 10.1523/JNEUROSCI.15-09-05859.1995 · DOI · Paper page
- Parthsarathy, 1992, Distributed but convergent ordering of corticostriatal projections: Analysis of the frontal eye field and the supplementary eye field in the macaque monkey, Journal of Neuroscience, 12, 4468, 10.1523/JNEUROSCI.12-11-04468.1992 · DOI · Paper page
- Picard, 2001, Imaging the premotor areas, Current Opinion in Neurobiology, 11, 663, 10.1016/S0959-4388(01)00266-5 · DOI · Paper page
- Price, 1996, Networks related to the orbital and medial prefrontal cortex; a substrate for emotional behavior?, Progress in Brain Research, 107, 523, 10.1016/S0079-6123(08)61885-3 · DOI · Paper page
- Schultz, 1992, Neuronal activity in monkey ventral striatum related to the expectation of reward, Journal of Neuroscience, 12, 4595, 10.1523/JNEUROSCI.12-12-04595.1992 · DOI · Paper page
- Selemon, 1985, Longitudinal topography and interdigitation of corticostriatal projections in the rhesus monkey, Journal of Neuroscience, 5, 776, 10.1523/JNEUROSCI.05-03-00776.1985 · DOI · Paper page
- Singh, S. P. (1992) Reinforcement learning with a hierarchy of an abstract models. In Proceedings of the tenth national conference on artificial intelligence (pp. 202–207)
- Sutton, 1999, Improved switching among temporally abstract actions, Vol. 11, 1066
- Takada, 1998, Corticostriatal projections from the somatic motor areas of the frontal cortex in the macaque monkey: Segregation versus overlap of input zones from the primary motor cortex, the supplementary motor area, and the premotor cortex, Experimental Brain Research, 120, 114, 10.1007/s002210050384 · DOI · Paper page
- Takikawa, 2004, A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping, Journal of Neurophysiology, 92, 2520, 10.1152/jn.00238.2004 · DOI · Paper page
- Talairach, J., & Tournoux, P. (1998). Co-planar stereotaxic atlas of the human brain. Thieme
Cited by
Bardo, M. T., & Hammer, R. P. (1991) · Neuroscience · doi:10.1016/0306-4522(91)90226-e · Cited by 3,219
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646
Metadata
- Authors
- Haruno, M., & Kawato, M.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2006-10
- Volume
- 19
- Issue
- 8
- Pages
- 1242-1254
- ISSN
- 0893-6080
- Cited by
- 129
- References
- 17
- Language
- en
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.