Learning, reward, and decision making
Open accessAbstract
In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems. We consider the nature of the interactions between these systems and show how these interactions can lead to either adaptive or maladaptive behavioral outcomes. We then review evidence that an additional system guides inference concerning the hidden states of other agents, such as their beliefs, preferences, and intentions, in a social context. We also describe emerging evidence for an arbitration mechanism between model-based and model-free reinforcement learning, placing such a mechanism within the broader context of the hierarchical control of behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For a model-free learning system, the transition from training to test session is irrelevant; the first trials of the test session in which no reward is experienced matter, initially eliciting large reward prediction errors that gradually reduce with each new reward-absent trial.
“For a model-free learning system, the transition from training to test session itself is irrelevant; it is the first trials of the test session in which no reward is experienced that matter. Those first trials initially elicit large reward prediction errors, which then gradually reduce with each new reward-absent trial.”
From Selection history: How reward modulates selectivity of visual attention. (2018) · cites this paper as O’Doherty, Cockburn, & Pauli, 2017
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThe connectivity from cortex to dorsomedial (caudate) and dorsolateral striatum (putamen) constitute parallel loops, namely the associative and motor loops.
“Previous studies proposed that the connectivity from cortex to dorsomedial (caudate) and dorsolateral striatum (putamen) constitute parallel loops, namely the associative and motor loops, respectively (Middleton, 2000, Milad and Rauch, 2012, O’Doherty et al., 2017, Seger, 2018, Yin and Knowlton, 2006).”
From Imbalance between the caudate and putamen connectivity in obsessive–compulsive disorder (2022) · cites this paper as O’Doherty et al., 2017
The habitual system simplifies decision-making by shifting to a stimulus-response mode of behavior, making decisions potentially value-less and inflexible but less cognitively demanding.
“With overtraining, the habitual system simplifies decision-making by shifting to a stimulus-response mode of behavior51, making decisions potentially value-less and inflexible but less cognitively demanding52.”
From Revealing abrupt transitions from goal-directed to habitual behavior (2026) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 201 · Cited by 475
Research works printed but not linked: 3
- Kuvayev L, Sutton R. 1996 Model-based RL with an approximate, learned model. Proc. Yale Worksh. Adapt. Learn. Syst., 9th, June 10–12, New Haven, CT, pp. 101– 5. New Haven, CT: Dunham Lab., Yale Univ.
- Sutton RS 1990 RL architectures for animats. Proc. Int. Conf. Simul. Adapt. Behav., 1st, From Animals to Animats, Cambridge, MA, pp. 288– 96. Cambridge, MA: MIT Press
- Thorndike EL. 1898 Animal intelligence: an experimental study of the associative processes in animals. Psychol. Rev. Monogr. Suppl 2(4): 1– 109
- Authors
- O'Doherty, J. P., Cockburn, J., & Pauli, W. M.
- Affiliations
- Division of Humanities and Social Sciences and Computation and Neural Systems Program, California Institute of Technology, Pasadena, California 91125;
- California Institute of Technology (US)
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2017-01-03
- Volume
- 68
- Issue
- 1
- Pages
- 73-100
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- PMC
- PMC6192677
- OpenAlex
- W2526753222
- Cited by
- 475
- References
- 201
- Citations by year
- 2026: 29 · 2025: 39 · 2024: 72 · 2023: 48 · 2022: 50 · 2021: 57 · 2020: 94 · 2019: 37 · 2018: 31 · 2017: 18
- FWCI
- 3.03
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- Funders
- National Institute of Mental Health (P50MH094258)
- National Institute on Drug Abuse (R01DA033077)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- model-based learning, model-free learning, goal-directed behavior, habitual behavior, operant conditioning, classical conditioning, reward, reinforcement learning, arbitration mechanism, social cognition, belief inference, maladaptive behavior
- MeSH terms
- Brain, Conditioning, Psychological, Decision Making, Goals, Humans, Learning, Neurosciences, Reinforcement, Psychology, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
