CoChat

Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning

Haruno, Masahiko, Kawato, Mitsuo · 2006 · Neural Networks
Cited by 129

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

The brain's most difficult computation in decision-making learning is searching for essential information related to rewards among vast multimodal inputs and then integrating it into beneficial behaviors. Contextual cues consisting of limbic, cognitive, visual, auditory, somatosensory, and motor signals need to be associated with both rewards and actions by utilizing an internal representation such as reward prediction and reward prediction error. Previous studies have suggested that a suitable brain structure for such integration is the neural circuitry associated with multiple cortico-striatal loops. However, computational exploration still remains into how the information in and around these multiple closed loops can be shared and transferred. Here, we propose a "heterarchical reinforcement learning" model, where reward prediction made by more limbic and cognitive loops is propagated to motor loops by spiral projections between the striatum and substantia nigra, assisted by cortical projections to the pedunculopontine tegmental nucleus, which sends excitatory input to the substantia nigra. The model makes several fMRI-testable predictions of brain activity during stimulus-action-reward association learning. The caudate nucleus and the cognitive cortical areas are correlated with reward prediction error, while the putamen and motor-related areas are correlated with stimulus-action-dependent reward prediction. Furthermore, a heterogeneous activity pattern within the striatum…

Overview

Integrity signals

Registry

Crossref · 10.1016/j.neunet.2006.06.007

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 21

  1. Alexander, 1990, Basal ganglia thalamocortical circuits: Parallel substrates for motor, oculomotor, “prefrontal” and “limbic” functions, Progress in Brain Research, 85, 119, 10.1016/S0079-6123(08)62678-3 · DOI · Paper page
  2. Barto, 1983, Neuron-like elements that can solve difficult learning control problems, IEEE Transactions on Systems Man and Cybernetics, 13, 835
  3. Brown, 1999, How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues, Journal of Neuroscience, 19, 10502, 10.1523/JNEUROSCI.19-23-10502.1999 · DOI · Paper page
  4. Dayan, 1993, Feudal reinforcement learning, Vol. 5, 271
  5. Friston, 1995, Statistical parametric maps in functional brain imaging: A general linear approach, Human Brain Mapping, 2, 189, 10.1002/hbm.460020402 · DOI · Paper page
  6. Gerardin, 2003, Foot, hand, face and eye representation in the human striatum, Cerebral Cortex, 13, 162, 10.1093/cercor/13.2.162 · DOI · Paper page
  7. Haruno, 2001, Mosaic model for sensorimotor learning and control, Neural Computation, 13, 2201, 10.1162/089976601750541778 · DOI · Paper page
  8. Haruno, 2006, Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning, Journal of Neurophysiology, 92, 948, 10.1152/jn.00382.2005 · DOI · Paper page
  9. Hollerman, 1998, Dopamine neurons report an error in the temporal prediction of reward during learning, Nature Neuroscience, 4, 304, 10.1038/1124 · DOI · Paper page
  10. Morimoto, 1999, Hierarchical reinforcement learning for motion learning: learning “stand-up” trajectories, Advanced Robotics, 13, 267, 10.1163/156855399X01314 · doi:10.1163/156855399x01314
  11. Oakman, 1995, Distribution of pontomesencephalic cholinergic neurons projecting to substantia nigra differs significantly from those projecting to ventral tegmental area, Journal of Neuroscience, 15, 5859, 10.1523/JNEUROSCI.15-09-05859.1995 · DOI · Paper page
  12. Parthsarathy, 1992, Distributed but convergent ordering of corticostriatal projections: Analysis of the frontal eye field and the supplementary eye field in the macaque monkey, Journal of Neuroscience, 12, 4468, 10.1523/JNEUROSCI.12-11-04468.1992 · DOI · Paper page
  13. Picard, 2001, Imaging the premotor areas, Current Opinion in Neurobiology, 11, 663, 10.1016/S0959-4388(01)00266-5 · DOI · Paper page
  14. Price, 1996, Networks related to the orbital and medial prefrontal cortex; a substrate for emotional behavior?, Progress in Brain Research, 107, 523, 10.1016/S0079-6123(08)61885-3 · DOI · Paper page
  15. Schultz, 1992, Neuronal activity in monkey ventral striatum related to the expectation of reward, Journal of Neuroscience, 12, 4595, 10.1523/JNEUROSCI.12-12-04595.1992 · DOI · Paper page
  16. Selemon, 1985, Longitudinal topography and interdigitation of corticostriatal projections in the rhesus monkey, Journal of Neuroscience, 5, 776, 10.1523/JNEUROSCI.05-03-00776.1985 · DOI · Paper page
  17. Singh, S. P. (1992) Reinforcement learning with a hierarchy of an abstract models. In Proceedings of the tenth national conference on artificial intelligence (pp. 202–207)
  18. Sutton, 1999, Improved switching among temporally abstract actions, Vol. 11, 1066
  19. Takada, 1998, Corticostriatal projections from the somatic motor areas of the frontal cortex in the macaque monkey: Segregation versus overlap of input zones from the primary motor cortex, the supplementary motor area, and the premotor cortex, Experimental Brain Research, 120, 114, 10.1007/s002210050384 · DOI · Paper page
  20. Takikawa, 2004, A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping, Journal of Neurophysiology, 92, 2520, 10.1152/jn.00238.2004 · DOI · Paper page
  21. Talairach, J., & Tournoux, P. (1998). Co-planar stereotaxic atlas of the human brain. Thieme

Cited by

See all 129

Metadata

Authors
Haruno, M., & Kawato, M.
Venue
Neural Networks
Publisher
Elsevier BV
Published
2006-10
Volume
19
Issue
8
Pages
1242-1254
ISSN
0893-6080
Cited by
129
References
17
Language
en
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.