Model‐based learning and the contribution of the orbitofrontal cortex to the model‐free world
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Learning is proposed to occur when there is a discrepancy between reward prediction and reward receipt. At least two separate systems are thought to exist: one in which predictions are proposed to be based on model‐free or cached values; and another in which predictions are model‐based. A basic neural circuit for model‐free reinforcement learning has already been described. In the model‐free circuit the ventral striatum (VS) is thought to supply a common‐currency reward prediction to midbrain dopamine neurons that compute prediction errors and drive learning. In a model‐based system, predictions can include more information about an expected reward, such as its sensory attributes or current, unique value. This detailed prediction allows for both behavioral flexibility and learning driven by changes in sensory features of rewards alone. Recent evidence from animal learning and human imaging suggests that, in addition to model‐free information, the VS also signals model‐based information. Further, there is evidence that the orbitofrontal cortex (OFC) signals model‐based information. Here we review these data and suggest that the OFC provides model‐based information to this traditional model‐free circuitry and offer possibilities as to how this interaction might occur.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 6×
“One such theory is temporal difference reinforcement learning (TDRL).”
BackgroundInfluentialDissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 2×
“activity in the ventral striatum (VS), consistent with that of a critic (Schultz et al., 1992; O’Doherty et al., 2004).”
BackgroundResultDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
Pavlovian conditioning: it's not what you think it is
Rescorla, R. A. (1988) · American Psychologist · doi:10.1037/0003-066x.43.3.151 · Cited by 1,542 · cited 1×
“A basic observation from studies of learning is that animals will adjust their behavior to reflect contingencies between events in their environment (Rescorla, 1988).”
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Motivational control of goal-directed action
Dickinson, A., & Balleine, B. (1994) · Animal Learning & Behavior · doi:10.3758/bf03199951 · Cited by 995
Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Day, J. J., Roitman, M. F., Wightman, R. M., & Carelli, R. M. (2007) · Nature Neuroscience · doi:10.1038/nn1923 · Cited by 710
BOLD responses reflecting dopaminergic signals in the human ventral tegmental area
D'Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008) · Science · doi:10.1126/science.1150605 · Cited by 661
Parallel incentive processing: an integrated view of amygdala function
Balleine, B. W., & Killcross, S. (2006) · Trends in Neurosciences · doi:10.1016/j.tins.2006.03.002 · Cited by 622
Cited by
Emotion and the prefrontal cortex: an integrative review
Dixon, M. L., Thiruchselvam, R., Todd, R., & Christoff, K. (2017) · Psychological Bulletin · doi:10.1037/bul0000096 · Cited by 745
What the orbitofrontal cortex does not do
Stalnaker, T. A., Cooch, N. K., & Schoenbaum, G. (2015) · Nature Neuroscience · doi:10.1038/nn.3982 · Cited by 531
Self-citationDecoding subjective decisions from orbitofrontal cortex
Rich, E. L., & Wallis, J. D. (2016) · Nature Neuroscience · doi:10.1038/nn.4320 · Cited by 371
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357
“Quite a wealth of evidence indicates that orbitofrontal and related prefrontal areas are involved in model-based predictions of the values associated with stimuli and their predictors, in cases in which value has been obtained by previous experiences in relevant states (Boorman, Behrens, Woolrich, & Rushworth, 2009; C…”
BackgroundBlanchard, T. C., Hayden, B. Y., & Bromberg-Martin, E. S. (2015) · Neuron · doi:10.1016/j.neuron.2014.12.050 · Cited by 295
Metadata
- Authors
- McDannald, M. A., Takahashi, Y. K., Lopatina, N., Pietras, B. W., Jones, J. L., & Schoenbaum, G.
- Venue
- European Journal of Neuroscience
- Publisher
- Wiley
- Published
- 2012-04
- Volume
- 35
- Issue
- 7
- Pages
- 991-996
- ISSN
- 0953-816X (ISSN-L)
- 1460-9568
- OpenAlex
- W2123419686
- Cited by
- 84
- References
- 28
- FWCI
- 5.37
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Model‐based learning and the contribution of the orbitofrontal cortex to the model‐free world. https://app.cochat.ai/papers/10.1111/j.1460-9568.2011.07982.x
- The paper
- McDannald, M. A., Takahashi, Y. K., Lopatina, N., Pietras, B. W., Jones, J. L., & Schoenbaum, G. (2012). Model‐based learning and the contribution of the orbitofrontal cortex to the model‐free world. European Journal of Neuroscience. https://doi.org/10.1111/j.1460-9568.2011.07982.x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.