CoChat
Sign Up
Cited by 500Neurotransmitter Receptor Influence on BehaviorNeural and Behavioral Psychology StudiesReinforcement Learning in Robotics

Actor–critic models of the basal ganglia: new anatomical and computational perspectives

Joel, Daphna, Niv, Yael, Ruppin, Eytan·2002·Neural Networks
DOI 10.1016/s0893-6080(02)00047-3
Open this paper in CoChat — free account

Abstract

A large number of computational models of information processing in the basal ganglia have been developed in recent years. Prominent in these are actor-critic models of basal ganglia functioning, which build on the strong resemblance between dopamine neuron activity and the temporal difference prediction error signal in the critic, and between dopamine-dependent long-term synaptic plasticity in the striatum and learning guided by a prediction error signal in the actor. We selectively review several actor-critic models of the basal ganglia with an emphasis on two important aspects: the way in which models of the critic reproduce the temporal dynamics of dopamine firing, and the extent to which models of the actor take into account known basal ganglia anatomy and physiology. To complement the efforts to relate basal ganglia mechanisms to reinforcement learning (RL), we introduce an alternative approach to modeling a critic network, which uses Evolutionary Computation techniques to 'evolve' an optimal RL mechanism, and relate the evolved mechanism to the basic model of the critic. We conclude our discussion of models of the critic by a critical discussion of the anatomical plausibility of implementations of a critic in basal ganglia circuitry, and conclude that such implementations build on assumptions that are inconsistent with the known anatomy of the basal ganglia. We return to the actor component of the actor-critic model, which is usually modeled at the striatal level with very little detail. We describe an alternative model of the basal ganglia which takes into account several important, and previously neglected, anatomical and physiological characteristics of basal ganglia-thalamocortical connectivity and suggests that the basal ganglia performs reinforcement-biased dimensionality reduction of cortical inputs. We further suggest that since such selective encoding may bias the representation at the level of the frontal cortex towards the selection of rewarded plans and actions, the reinforcement-driven dimensionality reduction framework may serve as a basis for basal ganglia actor models. We conclude with a short discussion of the dual role of the dopamine signal in RL and in behavioral switching.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/s0893-6080(02)00047-3

Cites

86 of 86 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Dopamine inputs to the striatum push the animal toward obtaining maximum reward, a process that must take into account possible dangers.

    “dopamine inputs to the striatum push the animal toward obtaining maximum reward, a process that must take into account possible dangers.”

    From Resolving the neural circuits of anxiety (2015) · cites this paper as [146]

  • Actor-critic models operate within the evaluation system.

    “Actor-critic models145-147 operate within the evaluation system”

    From Resolving the neural circuits of anxiety (2015) · cites this paper as [146]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Gerfen, C. R., Herkenham, M., & Thibault, J. (1987). The neostriatal mosaic II. Patchand matrixdirected mesostriatal dopaminergic and non-dopaminergic systems. Journal of Neuroscience , 7 , 3915-3934.
  2. Jaeger, D., Kita, H., & Wilson, C. J. (1994). Journal of Neurophysiology , 72 , 2555-2558.
  3. Oorschot, D. E. (1996). Total number of neurons in the neostriatal, pallidal, subthalamic, and substantia nigral nuclei of the rat basal ganglia: A stereological study using the cavalieri and optical disector methods. Journal of Comparative Neurology , 366 , 580-599.
  4. Rolls, E. T., & Johnstone, S. (1992). Neurophysiological analysis of striatal function. In C. Wallesch, & G. Vallar (Eds.), Neuropsychological disorders with subcortical lesions (pp. 61-97). Oxford: University Press.
  5. Sidibe, M., Bevan, M. D., Bolam, J. P., & Smith, Y. (1997). Efferent connections of the internal globus pallidus in the squirrel monkey. 1. Topography and synaptic organization of the pallidothalamic projection. Journal of Comparative Neurology , 382 , 323-347.
  6. Yelnik, J., Francois, C., Tand, D (1997). Proceedings if the Third Congress of European Neuroscience Society, Beaurdeax.

Cited by

Sorted by citations

Related

Authors
Joel, D., Niv, Y., & Ruppin, E.
Affiliations
Tel Aviv University (IL)
Corresponding author
Joel, Daphna
ORCID
Venue
Neural Networks
Publisher
Elsevier BV
Published
2002-06
Volume
15
Issue
4-6
Pages
535-547
ISSN
0893-6080 (ISSN-L)
DOI
10.1016/s0893-6080(02)00047-3
OpenAlex
W2128562176
Cited by
500
References
86
Citations by year
2026: 19 · 2025: 28 · 2024: 28 · 2023: 17 · 2022: 20 · 2021: 27 · 2020: 26 · 2019: 17 · 2018: 23 · 2017: 18 · 2016: 25 · 2015: 25 · 2014: 22 · 2013: 39 · 2012: 28
FWCI
2.80
Type
journal article
Language
en
Open access
closed
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Neural Networks (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
basal ganglia, actor-critic, dopamine, striatum, reinforcement learning, evolutionary computation, thalamocortical connections, dimensionality reduction, frontal cortex, switching behavior, synaptic plasticity
MeSH terms
Animals, Basal Ganglia, Humans, Learning, Models, Biological, Nerve Net

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review