The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
The ubiquity of model-based reinforcement learning
Open accessAbstract
The reward prediction error (RPE) theory of dopamine (DA) function has enjoyed great success in the neuroscience of learning and decision-making. This theory is derived from model-free reinforcement learning (RL), in which choices are made simply on the basis of previously realized rewards. Recently, attention has turned to correlates of more flexible, albeit computationally complex, model-based methods in the brain. These methods are distinguished from model-free learning by their evaluation of candidate actions using expected future outcomes according to a world model. Puzzlingly, signatures from these computations seem to be pervasive in the very same regions previously thought to support model-free learning. Here, we review recent behavioral and neural evidence about these two systems, in attempt to reconcile their enigmatic cohabitation in the brain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From an RL perspective, stress shifts the balance toward habits over lengthy planning by increasing experienced time pressure (Doll et al. 2012).”
From Psychology of Habit. (2016) · cites this paper as Doll et al. 2012
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given the profusion of direct and indirect connections between neural circuits underlying goal-directed and habitual (model-free) behaviors, goals can have a biasing influence on habit formation (Doll et al. 2012).”
From Psychology of Habit. (2016) · cites this paper as Doll et al. 2012
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1978) · UA Campus Repository (The University of Arizona) · Cited by 8,010
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
The reward circuit: linking primate anatomy and human imaging
Haber, S. N., & Knutson, B. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.129 · Cited by 3,718
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationThe Hippocampus as a Cognitive Map
Ranck, J. B. (1979) · American Journal of Psychiatry · doi:10.1176/ajp.136.10.1353 · Cited by 2,373
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Self-citationBalleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,381
Flagged first, then by citations
Cited by
Sorted by citations
Items 301–380 of 380
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0003 · Cited by 1
van Geen, C., Lempert, K. M., Cohen, M. S., MacNear, K. A., Reckers, F. M., Zaneski, L., Wolk, D. A., & Kable, J. W. (2026) · Journal of Neuroscience · doi:10.1523/jneurosci.1459-25.2026 · Cited by 1
Children leverage predictive representations for flexible, value-guided choice
Zhang, A., Kahn, A. E., Daw, N. D., Nussenbaum, K., & Hartley, C. A. (2025) · Cognition · doi:10.1016/j.cognition.2025.106340 · Cited by 1
Self-citationYang, C., Liu, T., Wen, M., & Liu, X. (2025) · npj Science of Learning · doi:10.1038/s41539-025-00363-w · Cited by 1
Learning and Using Different Task Rule Updating Strategies Require Extensive Practice
Xu, S., Wen, T., Egner, T., Verguts, T., & Braem, S. (2026) · Cognitive Science · doi:10.1111/cogs.70262 · Cited by 1
Hammouda, N. G., Khan, R., Mostafa, L., Musa, V. A., Rajab, H., Singh, N. S. S., Aich, W., & Hajlaoui, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-29997-x · Cited by 1
The prefrontal–striatal signatures of reduced model-based learning in depressed patients
Wang, X., Zhou, X., Zhang, D., Zhang, Z., Gong, Y., Feng, Z., & Hou, J. M. (2026) · Annals of General Psychiatry · doi:10.1186/s12991-026-00630-z · Cited by 1
Task, person, and experiential characteristics drive the transfer of learning
LaFollette, K. J., Frank, D. J., Burgoyne, A. P., & Macnamara, B. N. (2026) · Communications Psychology · doi:10.1038/s44271-026-00408-9 · Cited by 1
Prediction in action: Toward an empirical science of active inference
Lageman, J., Fahrenfort, J. J., & Slagter, H. A. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106817 · Cited by 1
Bayes Optimality of Human Perception, Action and Learning: Behavioural and Neural Evidence
Beierholm, U. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12084-3_10 · Cited by 1
Escitalopram Restores Reversal Learning Impairments in Rats with Lesions of Orbital Frontal Cortex
Tait, D. S., Bowman, E. E., Miller, S., Dovlatyan, M., Sánchez, C., & Brown, V. J. (2021) · Language, cognition and mind · doi:10.1007/978-3-030-50200-3_18 · Cited by 1
Kavaliotis, E., Mahlberg, J., Bennett, D., Verdejo-García, A., Ogeil, R. P., & Drummond, S. P. A. (2025) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2025.108072 · Cited by 1
A network for computing value homeostasis in the human medial prefrontal cortexPreprint
Juechems, K., Balaguer, J., Castañón, S. H., Ruz, M., O’Reilly, J. X., & Summerfield, C. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/278531 · Cited by 1
Griessinger, T., Coricelli, G., & Khamassi, M. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/346155 · Cited by 1
Modelo de Avaliação do Aprendizado de Metacompetências (MAAM) em cursos de Engenharias
Tiziotto, S. A. (2018) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.18.2018.tde-17072018-092243 · Cited by 1
Patrick, S., & Bullock, D. H. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/619445 · Cited by 2
Competitive Neurocognitive Functions Can Tell Us What to Learn and When: In Schools and BeyondPreprint
Németh, D., Schipper, K., Plèche, C., & Janacsek, K. (2020) · Preprints.org · doi:10.20944/preprints202006.0050.v1
Retrosplenial cortex is necessary for spatial and non-spatial latent learning in micePreprint
de Barros, A. C. B., Baruchin, L. J., Panayi, M. C., Nyberg, N., Samborska, V., Mealing, M. T., Akam, T., Kwag, J., Bannerman, D. M., & Kohl, M. M. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.07.21.453258 · Cited by 2
Editorial: On the Nature and Scope of Habits and Model-Free Control
Melnikoff, D. E., Bargh, J. A., & Wood, W. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.760841 · Cited by 1
Reward Bases: Instantaneous reward revaluation with temporal difference learningPreprint
Millidge, B., Walton, M. E., & Bogacz, R. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.14.488361 · Cited by 1
Investments and Asset Pricing in a World of Satisficing AgentsPreprint
Berrada, T., Bossaerts, P. L., & Ugazio, G. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4711883 · Cited by 1
Peng, Y., Okur, O., Lindersson, C., Tinelli, S., Stedehouder, J., Shah, R. S., Lak, A., Stagg, C. J., & Sharott, A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.04.621878 · Cited by 2
Transition ability to safe states reduces fear responses to heightPreprint
Haruno, M., & Fujino, M. (2024) · doi:10.31234/osf.io/r7p6w · Cited by 1
Brands, A. M. A., Knauth, K., Mathar, D., Lee, S., Kuzmanovic, B., Tittgemeyer, M., & Peters, J. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.18.665478 · Cited by 1
Meta-learning is expressed through altered prefrontal cortical dynamicsPreprint
Sun, X., Comrie, A. E., Kahn, A. E., Monroe, E. J., Washington, C. B., Joshi, A., Guidera, J. A., Denovellis, E. L., Krausz, T. A., Zhou, J., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.01.697272 · Cited by 1
Self-citationBasal Ganglia Regulation of Motivated Behaviors
Rossi, M. A. (2015) · DukeSpace (Duke University)
Exploiting Exploration: Past Outcomes and Future Actions
Louie, K. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.016
Action control in uncertain environments
Smittenaar, P. (2015) · UCL Discovery (University College London)
Computational Modeling of the Basal Ganglia : Functional Pathways and Reinforcement Learning
Berthet, P. (2015) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
One critic, two actors: Evidence for covert learning in the basal gangliaPreprint
Topalidou, M., Kase, D., Boraud, T., & Rougier, N. P. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/060236
The Biological Basis of Economic Choice
Freestone, D. M., & Balcı, F. (2017) · Wiley eBooks · doi:10.1002/9781118540770.ch7
Properties of decision-making tasks govern the tradeoff between model-based and model-free learningPreprint
Abzug, Z. M., Sommer, M. A., & Beck, J. M. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/730663
Adaptive Economics: A neuroethological approach to the study of preferences, biases, and choice
Bujold, P. M. (2019) · Apollo (University of Cambridge) · doi:10.17863/cam.48007
Combining Imagination and Heuristics to Learn Strategies that Generalize
Peterson, E. J., Müyesser, A., Verstynen, T., & Dunovan, K. (2020) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.13477
Planning with Expectation Models for ControlPreprint
Kudashkina, K., Wan, Y., Naik, A., & Sutton, R. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08543
The Journal of Neuroscience's 40th Anniversary: Looking Back, Looking Forward
Carew, T. J. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.0512-21.2021
Hamaguchi, K., Takahashi-Aoki, H., & Watanabe, D. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.12.27.474215
Stronger Prejudices Are Associated With Decreased Model-Based Control
Sebold, M., Chen, H., Önal, A., Kuitunen‐Paul, S., Mojtahedzadeh, N., Garbusow, M., Nebe, S., Wïttchen, H., Huys, Q. J. M., Schlagenhauf, F., et al. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.767022
On Becoming a Good Driver: Modeling the Learning of a Mental Model
Bhalwankar, R., & Jan Treur, A. (2022) · Studies in systems, decision and control · doi:10.1007/978-3-030-85821-6_3
In Control of Your Instructor: Modeling Learner-Controlled Mental Model Learning
Bhalwankar, R., & Jan Treur, A. (2022) · Studies in systems, decision and control · doi:10.1007/978-3-030-85821-6_9
Faculty Opinions recommendation of Habits without values.
Milton, A. L. (2021) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.734918397.793583295
Inferring Actions, Intentions, and Causal Relations in a Neural NetworkPreprint
Juechems, K., & Saxe, A. (2021) · doi:10.31234/osf.io/ptz2r
Vicarious learning: model-based or model-free?Preprint
Navidi, P., Saeedpour, S., Ershadmanesh, S., Hossein, M. M., & Bahrami, B. (2023) · doi:10.31234/osf.io/c3k64
Ten Tips for Running Behavioral Experiments Using Human ParticipantsPreprint
Konovalov, A. (2023) · doi:10.31234/osf.io/6t2sr
Learning a covert sequence of effector movements: limits to its acquisition
Johannsen, L., & Koch, I. (2023) · Psychological Research · doi:10.1007/s00426-023-01855-3
Tradeoffs in Uncertain Decision Making
Strickland, L. J. G., & Boag, R. J. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197501061.013.19
What’s Next if Reward is Enough? Insights for AGI from Animal Reinforcement Learning
Rajagopal, S. (2023) · Journal of Artificial General Intelligence · doi:10.2478/jagi-2023-0002
Yee, X. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qm3gp
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.002.0003
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0002
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.002.0004
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0007
Feelings Versus Representations of Value
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0005
Indicatives Versus Imperatives
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0006
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0008
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.002.0006
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0001
Traditional Critiques Critiqued
Carruthers, P. (2024) · Oxford University Press eBooks · doi:10.1093/oso/9780198906131.003.0004
Yu, C., & Kim, H. (2024) · Journal of the geological society of Korea · doi:10.14770/jgsk.2024.006
Cowan, E. T., & Murty, V. P. (2024) · Elsevier eBooks · doi:10.1016/b978-0-12-820480-1.00112-1
Modeling Multidimensional Cognitive Search in Creativity with Generalized Additive Model
Cheoh, J. L. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651267
Generalized cue reactivity in dopamine neurons after opioidsPreprint
Lehmann, C. M., Miller, N. E., Nair, V. S., Costa, K. M., Schoenbaum, G. M., & Moussawi, K. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.02.597025
Attention and learning strategies reveal distinct profiles of psychiatric traitsPreprint
Pettine, W. W., Tseng, A., Yang, A., Docherty, A. R., Redish, A. D., Murray, J. D., & Jacob, S. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/52mkw · Cited by 1
Predicting Asset Return Shifts: Experimental Evidence on Human Forecasting AbilityPreprint
Yang, Y., Payzan-LeNestour, É., & Tang, Q. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4935465
Individual Differences in Policy Precision: Links to Suicidal Ideation and Network DynamicsPreprint
Yoon, D., Kim, J., Kim, D. H., Shin, D. W., Bong, S. H., Kim, J., Park, H., Jeong, B., & Jeong, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.15.618415
Human hippocampal ripples prioritise model-based learningPreprint
Zhou, X., Wang, X., Hu, X., Wang, H., Zhang, J., Yu, Q., Xu, J., Xiao, Z., He, L., & Liu, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.31.667862
Reappraising Intuitions About ConsciousnessPreprint
Kim, C., & Chong, S. C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5420626
Cognition–Emotion Interactions
Armony, J., & Vuilleumier, P. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009342919.024
Neural Mechanisms of Arbitration between Multiple Experts in Value-Based Decision-Making
Grossman, C. D., & O’Doherty, J. P. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009342919.028
Striatal Value Signals Track Progress Toward a Goal During PlanningPreprint
Luzyanin, D., & Konovalov, A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.10.675322
Revisiting Self-Attention in Deep Reinforcement Learning: A Dimensional Perspective
Sun, J., Ong, D. P. J., & Lee, B. (2025) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai65422.2025.11189820
Practical Intelligent Automatic Train Operations which Optimize Switching Times Autonomously
Liu, X., Zhou, J., Men, Y., Pan, L., & Luo, S. (2025) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981251337670
Reappraising intuitions about consciousness
Kim, C., & Chong, S. C. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1747924
Computational learning noise in human decision-making
Findling, C. (2018) · theses.fr (ABES) · doi:10.70675/32e62279zea97z45b1z9326z06a449dae472
Impact of aging on human decision-making in ecological conditions
Aubert, J. D. S. (2023) · theses.fr (ABES) · doi:10.70675/f7143113zd412z4b5fza432z1032ef6ea209
Dück, K., Overmeyer, R., Dieterich, R., & Endraß, T. (2026) · Scientific Reports · doi:10.1038/s41598-026-58046-4
Dopamine projections to the basolateral amygdala enable reward predictionPreprint
Ramírez-Armenta, K., Feng, C., Martinez, A., Andrade, J. E., Quach, J., Paredes, N., Sias, A. C., Griffin, N. K., Sharpe, M. J., & Wassum, K. M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.21.733629
Delayed reward information is underweighted in reinforcement learning with dispersed feedback
Cotet, M., Poensgen, D., & Krajbich, I. M. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014459
Brands, A. M. A., Knauth, K., Mathar, D., Lee, S., Kuzmanovic, B., Tittgemeyer, M., & Peters, J. (2026) · Journal of Behavioral Addictions · doi:10.1556/2006.2025.00424 · Cited by 1
Computational Growth Modeling: Assessing Longitudinal Change with Joint ModelsPreprint
Haynes, J., Haines, N., Sullivan‐Toole, H., & Olino, T. (2026) · Research Square · doi:10.21203/rs.3.rs-11269872/v1
Metadata
- Authors
- Doll, B. B., Simon, D. A., & Daw, N. D.
- Affiliations
- Columbia University (US)
- New York University (US)
- Corresponding author
- Daw, Nathaniel D
- Venue
- Current Opinion in Neurobiology
- Publisher
- Elsevier BV
- Published
- 2012-12
- Volume
- 22
- Issue
- 6
- Pages
- 1075-1081
- ISSN
- 0959-4388 (ISSN-L)
- PMC
- PMC3513648
- OpenAlex
- W1977114987
- Cited by
- 380
- References
- 64
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 40 · 2023: 26 · 2022: 35 · 2021: 34 · 2020: 33 · 2019: 29 · 2018: 29 · 2017: 17 · 2016: 32 · 2015: 31 · 2014: 21 · 2013: 13
- FWCI
- 1.65
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Current Opinion in Neurobiology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- McKnight Foundation
- McDonnell Foundation
- NIMH (1R01MH087882-01)
- NINDS (1R01NS078784-01)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- model-based reinforcement learning, model-free reinforcement learning, reward prediction error, dopamine, decision-making, world models, neural correlates of learning, computational psychiatry
- MeSH terms
- Animals, Brain, Choice Behavior, Decision Making, Humans, Models, Neurological, Reinforcement, Psychology, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
