Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Behavior in spatial navigation is often organized into map-based (place-driven) vs. map-free (cue-driven) strategies; behavior in operant conditioning research is often organized into goal-directed vs. habitual strategies. Here we attempt to unify the two. We review one powerful theory for distinct forms of learning during instrumental conditioning, namely model-based (maintaining a representation of the world) and model-free (reacting to immediate stimuli) learning algorithms. We extend these lines of argument to propose an alternative taxonomy for spatial navigation, showing how various previously identified strategies can be distinguished as "model-based" or "model-free" depending on the usage of information and not on the type of information (e.g., cue vs. place). We argue that identifying "model-free" learning with dorsolateral striatum and "model-based" learning with dorsomedial striatum could reconcile numerous conflicting results in the spatial navigation literature. From this perspective, we further propose that the ventral striatum plays key roles in the model-building process. We propose that the core of the ventral striatum is positioned to learn the probability of action selection for every transition between states of the world. We further review suggestions that the ventral striatal core and shell are positioned to act as "critics" contributing to the computation of a reward prediction error for model-free and model-based systems, respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Spatial localization does not require the presence of local cues
Morris, R. G. M. (1981) · Learning and Motivation · doi:10.1016/0023-9690(81)90020-5 · Cited by 2,699
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 8×
“Our main hypothesis is that strategies for navigation, similar to strategies for instrumental conditioning (Daw et al., 2005), can be reconciled as either model-free or model-based—we define these terms below.”
BackgroundMethodInfluentialThe role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
From motivation to action: Functional interface between the limbic system and the motor system
MOGENSON, G., JONES, D., & YIM, C. (1980) · Progress in Neurobiology · doi:10.1016/0301-0082(80)90018-0 · Cited by 2,331 · cited 3×
“A general proposition is that the core is the route via which hippocampal sequencing of states reaches the motor system, a finessing of the long-recognized position of the core at the limbic-motor interface (Mogenson et al., 1980).”
BackgroundDissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088
Basal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 4×
“COMPUTATIONS BY THE STRIATUM Our proposed division of function between different parts of the striatum preserves the classical hypothesis that striatal territories all contribute to behavioral regulation but mainly differ in function because of their different afferents (Alexander et al., 1990; Joel and Weiner, 1994;…”
BackgroundInfluentialReverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727 · cited 1×
“Interestingly, sequences of hippocampal place cell activations that occur while an animal is running a track in search for reward are known to be replayed during subsequent sleep (Euston et al., 2007) or during awake resting periods (Foster and Wilson, 2006; Gupta et al., 2010).”
BackgroundLearning and Memory Functions of the Basal Ganglia
Packard, M. G., & Knowlton, B. J. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142937 · Cited by 1,717 · cited 1×
“In contrast, lesions of the DLS produce the opposite effect: impairing or reducing the expression of response strategies while sparing place strategies (Potegal, 1972; Devan and White, 1999; Adams et al., 2001; Packard and Knowlton, 2002; Martel et al., 2007).”
BackgroundHaber, S. N., Fudge, J. L., & McFarland, N. R. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-06-02369.2000 · Cited by 1,610
Cited by
The Nucleus Accumbens: An Interface Between Cognition, Emotion, and Action
Floresco, S. B. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010213-115159 · Cited by 938
Predictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
Dopamine neurons projecting to the posterior striatum form an anatomically distinct subclass
Menegas, W., Bergan, J. F., Ogawa, S. K., Isogai, Y., Umadevi Venkataraju, K., Osten, P., Uchida, N., & Watabe-Uchida, M. (2015) · eLife · doi:10.7554/elife.10032 · Cited by 322
Mannella, F., Gurney, K., & Baldassarre, G. (2013) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2013.00135 · Cited by 166
“Within this framework, a key role has been ascribed to the NAcc in terms of processing of current values and reward predictions (Humphries and Prescott, 2010; Bornstein and Daw, 2011; Penner and Mizumori, 2011; Pennartz et al., 2011; Khamassi and Humphries, 2012).”
A hippocampo-cerebellar centred network for the learning and execution of sequence-based navigation
Babayan, B. M., Watilliaux, A., Viejo, G., Paradis, A.-L., Girard, B., & Rondi-Reig, L. (2017) · Scientific Reports · doi:10.1038/s41598-017-18004-7 · Cited by 70
“The implication of the basal ganglia on one hand in sequence-based navigation, is compatible with human studies of route recognition and memory 29–31 , and orients the possible learning rules underlying acquisition of a sequence-based memory toward reinforcement learning 32,33 .”
Metadata
- Authors
- Khamassi, M., & Humphries, M. D.
- Venue
- Frontiers in Behavioral Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 79-79
- ISSN
- 1662-5153 (ISSN-L)
- OpenAlex
- W2124334449
- Cited by
- 123
- References
- 45
- FWCI
- 2.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.