Model‐based learning and the contribution of the orbitofrontal cortex to the model‐free world
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Learning is proposed to occur when there is a discrepancy between reward prediction and reward receipt. At least two separate systems are thought to exist: one in which predictions are proposed to be based on model‐free or cached values; and another in which predictions are model‐based. A basic neural circuit for model‐free reinforcement learning has already been described. In the model‐free circuit the ventral striatum (VS) is thought to supply a common‐currency reward prediction to midbrain dopamine neurons that compute prediction errors and drive learning. In a model‐based system, predictions can include more information about an expected reward, such as its sensory attributes or current, unique value. This detailed prediction allows for both behavioral flexibility and learning driven by changes in sensory features of rewards alone. Recent evidence from animal learning and human imaging suggests that, in addition to model‐free information, the VS also signals model‐based information. Further, there is evidence that the orbitofrontal cortex (OFC) signals model‐based information. Here we review these data and suggest that the OFC provides model‐based information to this traditional model‐free circuitry and offer possibilities as to how this interaction might occur.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 6×
“One such theory is temporal difference reinforcement learning (TDRL).”
BackgroundInfluentialDissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 2×
“activity in the ventral striatum (VS), consistent with that of a critic (Schultz et al., 1992; O’Doherty et al., 2004).”
BackgroundResultDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
Pavlovian conditioning: it's not what you think it is
Rescorla, R. A. (1988) · American Psychologist · doi:10.1037/0003-066x.43.3.151 · Cited by 1,542 · cited 1×
“A basic observation from studies of learning is that animals will adjust their behavior to reflect contingencies between events in their environment (Rescorla, 1988).”
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Motivational control of goal-directed action
Dickinson, A., & Balleine, B. (1994) · Animal Learning & Behavior · doi:10.3758/bf03199951 · Cited by 995
Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Day, J. J., Roitman, M. F., Wightman, R. M., & Carelli, R. M. (2007) · Nature Neuroscience · doi:10.1038/nn1923 · Cited by 710 · cited 2×
“A positive prediction error results in synchronous phasic DA release in both the DS and VS (Day et al., 2007; Roitman et al., 2008) to modulate learning, either directly or due to co-release of other agents (Lapish et al.”
BackgroundResultBOLD responses reflecting dopaminergic signals in the human ventral tegmental area
D'Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008) · Science · doi:10.1126/science.1150605 · Cited by 661
Parallel incentive processing: an integrated view of amygdala function
Balleine, B. W., & Killcross, S. (2006) · Trends in Neurosciences · doi:10.1016/j.tins.2006.03.002 · Cited by 622
Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597
Self-citationOrbitofrontal Cortex and Representation of Incentive Value in Associative Learning
Gallagher, M., McMahan, R. W., & Schoenbaum, G. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-15-06610.1999 · Cited by 593 · cited 1×
“, 2008) and reinforcer devaluation (Gallagher et al., 1999; Pickens et al., 2003, 2005; Izquierdo et al., 2004).”
BackgroundSelf-citationIzquierdo, A., Suda, R. K., & Murray, E. A. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.1921-04.2004 · Cited by 588
Corbit, L. H., Muir, J. L., & Balleine, B. W. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-09-03251.2001 · Cited by 564 · cited 2×
“This effect is dependent on the shell subregion of the VS (Corbit et al., 2001; Corbit & Balleine, 2011).”
BackgroundDopaminergic terminals in the nucleus accumbens but not the dorsal striatum corelease glutamate
Stuber, G. D., Hnasko, T. S., Britt, J. P., Edwards, R. H., & Bonci, A. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.1754-10.2010 · Cited by 529 · cited 1×
“, 2008) to modulate learning, either directly or due to co-release of other agents (Lapish et al., 2007; Stuber et al., 2010).”
BackgroundEveritt, B. J., Morris, K. A., O'Brien, A., & Robbins, T. W. (1991) · Neuroscience · doi:10.1016/0306-4522(91)90145-e · Cited by 507
Midbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · cited 1×
“actions evident in recent DA recording studies could also reflect model-based input (Morris et al., 2006; Bromberg-Martin et al., 2010; Xin & Costa, 2010).”
BackgroundDifferential control over cocaine-seeking behavior by nucleus accumbens core and shell
Ito, R., Robbins, T. W., & Everitt, B. J. (2004) · Nature Neuroscience · doi:10.1038/nn1217 · Cited by 501 · cited 1×
“, 1991), general-affective conditioned reinforcement (Ito et al., 2004), and general-affective Pavlovian to instrumental transfer (Corbit et al.”
BackgroundWyvell, C. L., & Berridge, K. C. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-19-07831.2001 · Cited by 436
Walton, M. E., Behrens, T. E. J., Buckley, M. J., Rudebeck, P. H., & Rushworth, M. F. S. (2010) · Neuron · doi:10.1016/j.neuron.2010.02.027 · Cited by 397
Orbitofrontal Cortex Mediates Outcome Encoding in Pavlovian But Not Instrumental Conditioning
Ostlund, S. B., & Balleine, B. W. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5443-06.2007 · Cited by 379
Hall, J., Parkinson, J. A., Connor, T. M., Dickinson, A., & Everitt, B. J. (2001) · European Journal of Neuroscience · doi:10.1046/j.0953-816x.2001.01577.x · Cited by 355
The temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310
Corbit, L. H., & Balleine, B. W. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2711-11.2011 · Cited by 309
Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280 · cited 1×
“recorded from putative DA neurons in the ventral tegmental area (VTA) in animals with ipsilateral sham or neurotoxic lesions of the OFC (Takahashi et al., 2011).”
BackgroundSelf-citationSetlow, B., Holland, P. C., & Gallagher, M. (2002) · Behavioral Neuroscience · doi:10.1037/0735-7044.116.2.267 · Cited by 160
Lex, B., & Hauber, W. (2010) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2009.11.002 · Cited by 61
Singh, T. (2010) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2010.00126 · Cited by 40 · cited 1×
“by VS lesions, falls into this category (Lex & Hauber, 2010; Singh et al., 2010).”
Background
Cited by
Emotion and the prefrontal cortex: an integrative review
Dixon, M. L., Thiruchselvam, R., Todd, R., & Christoff, K. (2017) · Psychological Bulletin · doi:10.1037/bul0000096 · Cited by 745
What the orbitofrontal cortex does not do
Stalnaker, T. A., Cooch, N. K., & Schoenbaum, G. (2015) · Nature Neuroscience · doi:10.1038/nn.3982 · Cited by 531
Self-citationDecoding subjective decisions from orbitofrontal cortex
Rich, E. L., & Wallis, J. D. (2016) · Nature Neuroscience · doi:10.1038/nn.4320 · Cited by 371
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357
“Quite a wealth of evidence indicates that orbitofrontal and related prefrontal areas are involved in model-based predictions of the values associated with stimuli and their predictors, in cases in which value has been obtained by previous experiences in relevant states (Boorman, Behrens, Woolrich, & Rushworth, 2009; C…”
BackgroundBlanchard, T. C., Hayden, B. Y., & Bromberg-Martin, E. S. (2015) · Neuron · doi:10.1016/j.neuron.2014.12.050 · Cited by 295
Metadata
- Authors
- McDannald, M. A., Takahashi, Y. K., Lopatina, N., Pietras, B. W., Jones, J. L., & Schoenbaum, G.
- Venue
- European Journal of Neuroscience
- Publisher
- Wiley
- Published
- 2012-04
- Volume
- 35
- Issue
- 7
- Pages
- 991-996
- ISSN
- 0953-816X (ISSN-L)
- 1460-9568
- OpenAlex
- W2123419686
- Cited by
- 84
- References
- 28
- FWCI
- 5.37
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.