The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
The ubiquity of model-based reinforcement learning
Open accessAbstract
The reward prediction error (RPE) theory of dopamine (DA) function has enjoyed great success in the neuroscience of learning and decision-making. This theory is derived from model-free reinforcement learning (RL), in which choices are made simply on the basis of previously realized rewards. Recently, attention has turned to correlates of more flexible, albeit computationally complex, model-based methods in the brain. These methods are distinguished from model-free learning by their evaluation of candidate actions using expected future outcomes according to a world model. Puzzlingly, signatures from these computations seem to be pervasive in the very same regions previously thought to support model-free learning. Here, we review recent behavioral and neural evidence about these two systems, in attempt to reconcile their enigmatic cohabitation in the brain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From an RL perspective, stress shifts the balance toward habits over lengthy planning by increasing experienced time pressure (Doll et al. 2012).”
From Psychology of Habit. (2016) · cites this paper as Doll et al. 2012
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given the profusion of direct and indirect connections between neural circuits underlying goal-directed and habitual (model-free) behaviors, goals can have a biasing influence on habit formation (Doll et al. 2012).”
From Psychology of Habit. (2016) · cites this paper as Doll et al. 2012
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1978) · UA Campus Repository (The University of Arizona) · Cited by 8,010
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
The reward circuit: linking primate anatomy and human imaging
Haber, S. N., & Knutson, B. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.129 · Cited by 3,718
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationThe Hippocampus as a Cognitive Map
Ranck, J. B. (1979) · American Journal of Psychiatry · doi:10.1176/ajp.136.10.1353 · Cited by 2,373
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Self-citationBalleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Flagged first, then by citations
Cited by
Sorted by citations
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 871
Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J. R., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 460
Oldham, S., Murawski, C., Fornito, A., Youssef, G. J., Yücel, M., & Lorenzetti, V. (2018) · Human Brain Mapping · doi:10.1002/hbm.24184 · Cited by 449
Metadata
- Authors
- Doll, B. B., Simon, D. A., & Daw, N. D.
- Affiliations
- Columbia University (US)
- New York University (US)
- Corresponding author
- Daw, Nathaniel D
- Venue
- Current Opinion in Neurobiology
- Publisher
- Elsevier BV
- Published
- 2012-12
- Volume
- 22
- Issue
- 6
- Pages
- 1075-1081
- ISSN
- 0959-4388 (ISSN-L)
- PMC
- PMC3513648
- OpenAlex
- W1977114987
- Cited by
- 380
- References
- 64
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 40 · 2023: 26 · 2022: 35 · 2021: 34 · 2020: 33 · 2019: 29 · 2018: 29 · 2017: 17 · 2016: 32 · 2015: 31 · 2014: 21 · 2013: 13
- FWCI
- 1.65
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Current Opinion in Neurobiology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- McKnight Foundation
- McDonnell Foundation
- NIMH (1R01MH087882-01)
- NINDS (1R01NS078784-01)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- model-based reinforcement learning, model-free reinforcement learning, reward prediction error, dopamine, decision-making, world models, neural correlates of learning, computational psychiatry
- MeSH terms
- Animals, Brain, Choice Behavior, Decision Making, Humans, Models, Neurological, Reinforcement, Psychology, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
