The ubiquity of model-based reinforcement learning

Doll, Bradley B; Simon, Dylan A; Daw, Nathaniel D

doi:10.1016/j.conb.2012.08.003

Cited by 380Neural and Behavioral Psychology StudiesNeurotransmitter Receptor Influence on BehaviorReinforcement Learning in Robotics

The ubiquity of model-based reinforcement learning

Open access

Doll, Bradley B, Simon, Dylan A, Daw, Nathaniel D·2012·Current Opinion in Neurobiology

DOI 10.1016/j.conb.2012.08.003PMC PMC3513648

Open this paper in CoChat — free account

Abstract

The reward prediction error (RPE) theory of dopamine (DA) function has enjoyed great success in the neuroscience of learning and decision-making. This theory is derived from model-free reinforcement learning (RL), in which choices are made simply on the basis of previously realized rewards. Recently, attention has turned to correlates of more flexible, albeit computationally complex, model-based methods in the brain. These methods are distinguished from model-free learning by their evaluation of candidate actions using expected future outcomes according to a world model. Puzzlingly, signatures from these computations seem to be pervasive in the very same regions previously thought to support model-free learning. Here, we review recent behavioral and neural evidence about these two systems, in attempt to reconcile their enigmatic cohabitation in the brain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.conb.2012.08.003

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

From an RL perspective, stress shifts the balance toward habits over lengthy planning by increasing experienced time pressure.
“From an RL perspective, stress shifts the balance toward habits over lengthy planning by increasing experienced time pressure (Doll et al. 2012).”
From Psychology of Habit. (2016) · cites this paper as Doll et al. 2012
In “From goals and habits...” (page unavailable), the paper says the time cost of model-based evaluation may explain why chronic stress in rats and acute stress in humans shift behavior toward habits. It does not say that stress increases experienced time pressure, and it presents the time-cost explanation as a possibility rather than an established mechanism. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Given the profusion of direct and indirect connections between neural circuits underlying goal-directed and habitual behaviors, goals can have a biasing influence on habit formation.
“Given the profusion of direct and indirect connections between neural circuits underlying goal-directed and habitual (model-free) behaviors, goals can have a biasing influence on habit formation (Doll et al. 2012).”
From Psychology of Habit. (2016) · cites this paper as Doll et al. 2012
In “The way forward” (page unavailable), the paper describes a proposal that explicit representations can directly bias a model-free learner, supporting the possibility that goals influence habit learning. It does not establish the claimed profusion of direct and indirect connections between the neural circuits as the reason for that influence. (LLM, conf=0.82)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Untitled work
(1978) · UA Campus Repository (The University of Arizona) · Cited by 8,010 · see citing works
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641 · see citing works
The reward circuit: linking primate anatomy and human imaging
Haber, S. N., & Knutson, B. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.129 · Cited by 3,718 · see citing works
Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Self-citation
The Hippocampus as a Cognitive Map
Ranck, J. B. (1979) · American Journal of Psychiatry · doi:10.1176/ajp.136.10.1353 · Cited by 2,373 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Self-citation
Human and Rodent Homologies in Action Control: Corticostriatal Determinants of Goal-Directed and Habitual Action
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775 · see citing works
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598 · see citing works
Two types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378 · see citing works
Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
Phasic Firing in Dopaminergic Neurons Is Sufficient for Behavioral Conditioning
Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., & Deisseroth, K. (2009) · Science · doi:10.1126/science.1168878 · Cited by 1,266 · see citing works
Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226 · see citing works
States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166 · see citing works
Self-citation
Animal Intelligence: An Experimental Study of the Associative Processes in Animals
Kline, L. W., & Thorndike, E. L. (1898) · The American Journal of Psychology · doi:10.2307/1412682 · Cited by 1,110 · see citing works
The role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099 · see citing works
Chronic Stress Causes Frontostriatal Reorganization and Affects Decision-Making
Dias-Ferreira, E., Sousa, J. C., Melo, I., Morgado, P., Mesquita, A. R., Cerqueira, J. J., Costa, R. M., & Sousa, N. (2009) · Science · doi:10.1126/science.1171203 · Cited by 858 · see citing works
Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli
Brischoux, F., Chakraborty, S., Brierley, D. I., & Ungless, M. A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0811507106 · Cited by 801 · see citing works
Animal intelligence: An experimental study of the associative processes in animals.
Thorndike, E. L. (1898) · The Psychological Monographs · doi:10.1037/h0092987 · Cited by 795 · see citing works
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · see citing works
What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?
Doya, K. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00046-5 · Cited by 739 · see citing works
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · see citing works
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 635 · see citing works
Determining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529 · see citing works
The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498 · see citing works
Adaptive Critics and the Basal Ganglia
Barto, A. G. (1994) · The MIT Press eBooks · doi:10.7551/mitpress/4708.003.0018 · Cited by 466 · see citing works
Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees
Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002410 · Cited by 439 · see citing works
Are you or aren’t you? Challenges associated with physiologically identifying dopamine neurons
Ungless, M. A., & Grace, A. A. (2012) · Trends in Neurosciences · doi:10.1016/j.tins.2012.02.003 · Cited by 431 · see citing works
Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior
Zweifel, L. S., Parker, J. G., Lobb, C. J., Rainwater, A., Wall, V. Z., Fadok, J. P., Darvas, M., Kim, M. J., Mizumori, S. J. Y., Paladini, C. A., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0813415106 · Cited by 426 · see citing works
Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation
Faure, A., Haberland, U., Condé, F., & Massioui, N. E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3894-04.2005 · Cited by 402 · see citing works
Cross-species studies of orbitofrontal cortex and value-based decision-making
Wallis, J. D. (2011) · Nature Neuroscience · doi:10.1038/nn.2956 · Cited by 356 · see citing works
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Mechanisms of Hierarchical Reinforcement Learning in Corticostriatal Circuits 1: Computational Analysis
Frank, M. J., & Badre, D. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhr114 · Cited by 330 · see citing works
Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists.
Dickinson, A., Smith, J. W., & Mirenowicz, J. (2000) · Behavioral Neuroscience · doi:10.1037//0735-7044.114.3.468 · Cited by 312 · see citing works
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310 · see citing works
Shift from goal-directed to habitual cocaine seeking after prolonged experience in rats
Zapata, A., Minney, V. L., & Shippenberg, T. S. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4072-10.2010 · Cited by 296 · see citing works
Stress-induced modulation of instrumental behavior: From goal-directed to habitual control of action
Schwabe, L., & Wolf, O. T. (2011) · Behavioural Brain Research · doi:10.1016/j.bbr.2010.12.038 · Cited by 270 · see citing works
Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning
McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. M. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.5499-10.2011 · Cited by 215 · see citing works
Triple Dissociation of Information Processing in Dorsal Striatum, Ventral Striatum, and Hippocampus on a Learned Spatial Decision Task
van der Meer, M. A. A., Johnson, A. C., Schmitzer-Torbert, N. C., & Redish, A. D. (2010) · Neuron · doi:10.1016/j.neuron.2010.06.023 · Cited by 210 · see citing works
Instructional control of reinforcement learning: A behavioral and neurocomputational investigation
Doll, B. B., Jacobs, W. J., Sanfey, A. G., & Frank, M. J. (2009) · Brain Research · doi:10.1016/j.brainres.2009.07.007 · Cited by 209 · see citing works
Self-citation
Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans
de Wit, S., Corlett, P. R., Aitken, M. R., Dickinson, A., & Fletcher, P. C. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1639-09.2009 · Cited by 200 · see citing works
Socially evaluated cold pressor stress after instrumental learning favors habits over goal-directed action
Schwabe, L., & Wolf, O. T. (2010) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2009.12.010 · Cited by 186 · see citing works
Dopaminergic Genes Predict Individual Differences in Susceptibility to Confirmation Bias
Doll, B. B., Hutchison, K. E., & Frank, M. J. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.6486-10.2011 · Cited by 184 · see citing works
Self-citation
A Pallidus-Habenula-Dopamine Pathway Signals Inferred Stimulus Values
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., & Hikosaka, O. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00158.2010 · Cited by 182 · see citing works
Neural Correlates of Forward Planning in a Spatial Decision Task in Humans
Simon, D. A., & Daw, N. D. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.4647-10.2011 · Cited by 181 · see citing works
Self-citation
Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates.
Solway, A., & Botvinick, M. (2012) · Psychological Review · doi:10.1037/a0026435 · Cited by 180 · see citing works
Distributed Coding of Actual and Hypothetical Outcomes in the Orbital and Dorsolateral Prefrontal Cortex
Abe, H., & Lee, D. (2011) · Neuron · doi:10.1016/j.neuron.2011.03.026 · Cited by 168 · see citing works
The Neural Basis of Following Advice
Biele, G., Rieskamp, J., Krugel, L. K., & Heekeren, H. R. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1001089 · Cited by 157 · see citing works
Proceedings of the seventh international conference (1990) on Machine learning
Morgan, M. B. (1990) · Cited by 140 · see citing works
NMDA Receptors in Dopaminergic Neurons Are Crucial for Habit Learning
Wang, L. P., Li, F. R., Wang, D., Xie, K., Wang, D., Shen, X., & Tsien, J. Z. (2011) · Neuron · doi:10.1016/j.neuron.2011.10.019 · Cited by 122 · see citing works
Dissociating hippocampal and striatal contributions to sequential prediction learning
Bornstein, A. M., & Daw, N. D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07920.x · Cited by 120 · see citing works
Self-citation
Habitual versus goal-directed action control in Parkinson disease
de Wit, S., Barker, R. A., Dickinson, A. D., & Cools, R. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21514 · Cited by 116 · see citing works
Generalization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · see citing works
Self-citation
Instrumental uncertainty as a determinant of behavior under interval schedules of reinforcement
DeRusso, A. L. (2010) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2010.00017 · Cited by 111 · see citing works
Integrating Sample-Based Planning and Model-Based Reinforcement Learning
Walsh, T., Goschin, S., & Littman, M. L. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7689 · Cited by 105 · see citing works
Reliance on habits at the expense of goal-directed control following dopamine precursor depletion
de Wit, S., Standing, H. R., DeVito, E. E., Robinson, O. J., Ridderinkhof, K. R., Robbins, T. W., & Sahakian, B. J. (2012) · Psychopharmacology · doi:10.1007/s00213-011-2563-2 · Cited by 105 · see citing works
Reward prediction based on stimulus categorization in primate lateral prefrontal cortex
Pan, X., Sawa, K., Tsuda, I., Tsukada, M., & Sakagami, M. (2008) · Nature Neuroscience · doi:10.1038/nn.2128 · Cited by 84 · see citing works
Expectancies in decision making, reinforcement learning, and ventral striatum
van der Meer, M. A. A. (2010) · Frontiers in Neuroscience · doi:10.3389/neuro.01.006.2010 · Cited by 67 · see citing works
Models of Information processing in the Basal Ganglia
Mitchell, I. (1996) · Journal of Chemical Neuroanatomy · doi:10.1016/0891-0618(96)00155-x · Cited by 63 · see citing works
Separate encoding of model-based and model-free valuations in the human brain
Beierholm, U., Anen, C., Quartz, S. R., & Bossaerts, P. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.071 · Cited by 41 · see citing works
Hedging Your Bets by Learning Reward Correlations in the Human Brain
Wunderlich, K., Symmonds, M., Bossaerts, P. L., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.07.025 · Cited by 38 · see citing works
Evidence for model-based action planning in a sequential finger movement task
Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010) · Journal of Motor Behavior · doi:10.1080/00222895.2010.526467 · Cited by 36 · see citing works
Environmental statistics and the trade-off between model-based and TD learning in humans
Simon, D. A., & Daw, N. D. (2011) · neural information processing systems · Cited by 23 · see citing works
Self-citation
Dual-System Learning Models and Drugs of Abuse
Simon, D. A., & Daw, N. D. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0751-5_5 · Cited by 20 · see citing works
Self-citation

Flagged first, then by citations

Research works printed but not linked: 1

[41]Bromberg-Martin ES Matsumoto M Hong S Hikosaka O A pallidushabenula-dopamine pathway signals inferred stimulus values J Neurophysiol 2010 104 2 1068 1076 20538770 PMC2934919

Sorted by citations

Psychology of Habit.
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445 · see citing works
Goals and Habits in the Brain
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 668 · see citing works
Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J. R., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 430 · see citing works
The anticipation and outcome phases of reward and loss processing: A neuroimaging meta‐analysis of the monetary incentive delay task
Oldham, S., Murawski, C., Fornito, A., Youssef, G. J., Yücel, M., & Lorenzetti, V. (2018) · Human Brain Mapping · doi:10.1002/hbm.24184 · Cited by 425 · see citing works
Mechanisms of motivation–cognition interaction: challenges and opportunities
Braver, T. S., Krug, M. K., Chiew, K. S., Kool, W., Westbrook, J. A., Clement, N. J., Adcock, R. A., Barch, D. M., Botvinick, M., Carver, C. S., et al. (2014) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-014-0300-0 · Cited by 400 · see citing works
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357 · see citing works
The basolateral amygdala in reward learning and addiction
Wassum, K. M., & Izquierdo, A. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.08.017 · Cited by 306 · see citing works
Dysfunctions of decision‐making and cognitive control as transdiagnostic mechanisms of mental disorders: advances, gaps, and needs in current research
Goschke, T. (2014) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1410 · Cited by 293 · see citing works
Reinforcement learning in artificial and biological systems
Neftci, E., & Averbeck, B. B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0025-4 · Cited by 303 · see citing works
Modeling, diagnostics, optimization, and control of internal combustion engines via modern machine learning techniques: A review and future directions
Aliramezani, M., Koch, C. R., & Shahbakhti, M. (2021) · Progress in Energy and Combustion Science · doi:10.1016/j.pecs.2021.100967 · Cited by 293 · see citing works
Computational approaches to psychiatry
Stephan, K. E., & Mathys, C. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2013.12.007 · Cited by 260 · see citing works
The drift diffusion model as the choice rule in reinforcement learning
Pedersen, M. L., Frank, M. J., & Biele, G. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1199-y · Cited by 272 · see citing works
Model-based choices involve prospective neural activity
Doll, B. B., Duncan, K., Simon, D. A., Shohamy, D., & Daw, N. (2015) · Nature Neuroscience · doi:10.1038/nn.3981 · Cited by 284 · see citing works
Self-citation
Habits without values.
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280 · see citing works
From Creatures of Habit to Goal-Directed Learners
Decker, J. H., Otto, A. R., Daw, N. D., & Hartley, C. A. (2016) · Psychological Science · doi:10.1177/0956797616639301 · Cited by 265 · see citing works
Self-citation
Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making
Deserno, L., Huys, Q. J. M., Boehme, R., Buchert, R., Heinze, H.-J., Grace, A. A., Dolan, R. J., Heinz, A., & Schlagenhauf, F. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1417219112 · Cited by 233 · see citing works
Reinforcement learning in depression: a review of computational research
Chen, C., Takahashi, T., Nakagawa, S., Inoue, T., & Kusumi, I. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.05.005 · Cited by 233 · see citing works
Executive control and decision-making in the prefrontal cortex
Domenech, P., & Koechlin, É. (2014) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.10.007 · Cited by 237 · see citing works
Acute stress and episodic memory retrieval: neurobiological mechanisms and behavioral consequences
Gagnon, S. A., & Wagner, A. D. (2016) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.12996 · Cited by 233 · see citing works
Mechanisms Underlying Motivational Deficits in Psychopathology: Similarities and Differences in Depression and Schizophrenia
Barch, D. M., Pagliaccio, D., & Luking, K. (2015) · Current Topics in Behavioral Neurosciences · doi:10.1007/7854_2015_376 · Cited by 218 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Dynamics of awake hippocampal-prefrontal replay for spatial learning and memory-guided decision making
Shin, J. D., Tang, W., & Jadhav, S. P. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.012 · Cited by 199 · see citing works
Behaviour Centred Design: towards an applied science of behaviour change
Aunger, R. V., & Curtis, V. A. (2016) · Health Psychology Review · doi:10.1080/17437199.2016.1219673 · Cited by 223 · see citing works
VTA neurons coordinate with the hippocampal reactivation of spatial experience
Gomperts, S. N., Kloosterman, F., & Wilson, M. A. (2015) · eLife · doi:10.7554/elife.05360 · Cited by 191 · see citing works
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195 · see citing works
Self-citation
Attention as an effect not a cause
Krauzlis, R. J., Bollimunta, A., Arcizet, F., & Wang, L. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.05.008 · Cited by 193 · see citing works
Abstract context representations in primate amygdala and prefrontal cortex
Saez, A., Rigotti, M., Ostojic, S., Fusi, S., & Salzman, C. D. (2015) · Neuron · doi:10.1016/j.neuron.2015.07.024 · Cited by 183 · see citing works
Learning to minimize efforts versus maximizing rewards: computational principles and neural correlates
Skvortsova, V., Palminteri, S., & Pessiglione, M. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1350-14.2014 · Cited by 177 · see citing works
Decision Making: From Neuroscience to Psychiatry
Lee, D. (2013) · Neuron · doi:10.1016/j.neuron.2013.04.008 · Cited by 169 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
Humans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153 · see citing works
Model-based hierarchical reinforcement learning and human action control
Botvinick, M., & Weinstein, A. (2014) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2013.0480 · Cited by 153 · see citing works
The Computational Development of Reinforcement Learning during Adolescence
Palminteri, S., Kilford, E. J., Coricelli, G., & Blakemore, S. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004953 · Cited by 152 · see citing works
The role of experience in adolescent cognitive development: Integration of executive, memory, and mesolimbic systems
Murty, V. P., Calabro, F. J., & Luna, B. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.07.034 · Cited by 152 · see citing works
Instrumental learning of traits versus rewards: dissociable neural correlates and effects on choice
Hackel, L. M., Doll, B. B., & Amodio, D. M. (2015) · Nature Neuroscience · doi:10.1038/nn.4080 · Cited by 148 · see citing works
Self-citation
Cortical delta activity reflects reward prediction error and related behavioral adjustments, but at different times
Cavanagh, J. F. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.02.007 · Cited by 148 · see citing works
Episodic memories predict adaptive value-based decision-making.
Murty, V. P., FeldmanHall, O., Hunter, L. E., Phelps, E. A., & Davachi, L. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000158 · Cited by 140 · see citing works
What Are Memories For? The Hippocampus Bridges Past Experience with Future Decisions
Biderman, N., Bakkour, A., & Shohamy, D. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.04.004 · Cited by 147 · see citing works
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 137 · see citing works
Taking stock of value in the orbitofrontal cortex
Knudsen, E. B., & Wallis, J. D. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00589-2 · Cited by 137 · see citing works
Of goals and habits: age-related and individual differences in goal-directed decision-making
Eppinger, B., Walter, M., Heekeren, H. R., & Li, S. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00253 · Cited by 131 · see citing works
Moral Learning: Conceptual foundations and normative relevance
Railton, P. (2016) · Cognition · doi:10.1016/j.cognition.2016.08.015 · Cited by 130 · see citing works
Gaze-informed, task-situated representation of space in primate hippocampus during virtual navigation
Wirth, S., Baraduc, P., Planté, A., Pinède, S., & Duhamel, J.-R. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2001045 · Cited by 129 · see citing works
Dopamine selectively remediates ‘model-based’ reward learning: a computational approach
Sharp, M. E., Foerde, K., Daw, N. D., & Daphna Shohamy, A. (2015) · Brain · doi:10.1093/brain/awv347 · Cited by 129 · see citing works
Self-citation
Beyond dichotomies in reinforcement learning
Collins, A. G. E., & Cockburn, J. (2020) · Nature reviews. Neuroscience · doi:10.1038/s41583-020-0355-6 · Cited by 128 · see citing works
Reinforcement-learning in fronto-striatal circuits
Averbeck, B. B., & O’Doherty, J. P. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01108-0 · Cited by 127 · see citing works
Lateral orbitofrontal cortex anticipates choices and integrates prior with current information
Nogueira, R., Abolafia, J. M., Drugowitsch, J., Balaguer‐Ballester, E., Sanchez-Vives, M. V., & Moreno‐Bote, R. (2017) · Nature Communications · doi:10.1038/ncomms14823 · Cited by 124 · see citing works
Computational Rationality as a Theory of Interaction
Oulasvirta, A., Jokinen, J., & Howes, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517739 · Cited by 128 · see citing works
Prefrontal Cortex Predicts State Switches during Reversal Learning
Bartolo, R., & Averbeck, B. B. (2020) · Neuron · doi:10.1016/j.neuron.2020.03.024 · Cited by 123 · see citing works
Gaussian Processes for Learning and Control: A Tutorial with Examples
Liu, M., Chowdhary, G. V., da Silva, B. C., Liu, S.-Y., & How, J. P. (2018) · IEEE Control Systems · doi:10.1109/mcs.2018.2851010 · Cited by 121 · see citing works
Habitual versus goal-directed action control in Parkinson disease
de Wit, S., Barker, R. A., Dickinson, A. D., & Cools, R. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21514 · Cited by 116 · see citing works
Learning action-oriented models through active inference
Tschantz, A., Seth, A. K., & Buckley, C. L. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007805 · Cited by 121 · see citing works
The role of the orbitofrontal cortex in alcohol use, abuse, and dependence
Moorman, D. E. (2018) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2018.01.010 · Cited by 109 · see citing works
Closed-Loop Theta Stimulation in the Orbitofrontal Cortex Prevents Reward-Based Learning
Knudsen, E. B., & Wallis, J. D. (2020) · Neuron · doi:10.1016/j.neuron.2020.02.003 · Cited by 112 · see citing works
Learning and Animal Movement
Lewis, M. A., Fagan, W. F., Auger‐Méthé, M., Frair, J. L., Fryxell, J. M., Gros, C., Gurarie, E. D., Healy, S. D., & Merkle, J. A. (2021) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2021.681704 · Cited by 112 · see citing works
Models, movements, and minds: bridging the gap between decision making and action
Wispinski, N. J., Gallivan, J. P., & Chapman, C. S. (2018) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13973 · Cited by 105 · see citing works
Using fMRI to study reward processing in humans: past, present, and future
Wang, K. S., Smith, D. V., & Delgado, M. R. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00333.2015 · Cited by 105 · see citing works
Neural Basis of Strategic Decision Making
Lee, D., & Seo, H. (2015) · Trends in Neurosciences · doi:10.1016/j.tins.2015.11.002 · Cited by 104 · see citing works
Variability in Dopamine Genes Dissociates Model-Based and Model-Free Reinforcement Learning
Doll, B. B., Bath, K. G., Daw, N. D., & Frank, M. J. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.1901-15.2016 · Cited by 105 · see citing works
Self-citation
Species Distribution Modeling for Machine Learning Practitioners: A Review
Beery, S., Cole, E., Parker, J., Perona, P., & Winner, K. (2021) · ACM SIGCAS Conference on Computing and Sustainable Societies · doi:10.1145/3460112.3471966 · Cited by 100 · see citing works
Decoding cognition from spontaneous neural activity
Liu, Y., Nour, M. M., Schuck, N. W., Behrens, T. E. J., & Dolan, R. J. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00570-z · Cited by 100 · see citing works
The structure of reinforcement-learning mechanisms in the human brain
O’Doherty, J. P., Lee, S. W., & McNamee, D. (2014) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.10.004 · Cited by 97 · see citing works
Identity prediction errors in the human midbrain update reward-identity expectations in the orbitofrontal cortex
Howard, J. D., & Kahnt, T. (2018) · Nature Communications · doi:10.1038/s41467-018-04055-5 · Cited by 96 · see citing works
Reflections on dynamics, adaptation and control: A cognitive architecture for mental models
van Ments, L., & Jan Treur, A. (2021) · Cognitive Systems Research · doi:10.1016/j.cogsys.2021.06.004 · Cited by 95 · see citing works
Multiple memory systems as substrates for multiple decision systems
Doll, B. B., Daphna Shohamy, A., & Daw, N. D. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.04.014 · Cited by 92 · see citing works
Self-citation
Explicit and implicit reinforcement learning across the psychosis spectrum.
Barch, D. M., Carter, C. S., Gold, J. M., Johnson, S. L., Kring, A. M., MacDonald, A. W., Pizzagalli, D. A., Ragland, J. D., Silverstein, S. M., & Strauss, M. E. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000259 · Cited by 93 · see citing works
Measuring reinforcement learning and motivation constructs in experimental animals: Relevance to the negative symptoms of schizophrenia
Markou, A., Salamone, J. D., Bussey, T. J., Mar, A. C., Brunner, D., Gilmour, G., & Balsam, P. D. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.08.007 · Cited by 93 · see citing works
A theory of actions and habits: The interaction of rate correlation and contiguity systems in free-operant behavior.
Perez, O. D., & Dickinson, A. (2020) · Psychological Review · doi:10.1037/rev0000201 · Cited by 92 · see citing works
Learning to predict or predicting to learn?
Rabagliati, H., Gambi, C., & Pickering, M. J. (2015) · Language Cognition and Neuroscience · doi:10.1080/23273798.2015.1077979 · Cited by 90 · see citing works
Gaze data reveal distinct choice processes underlying model-based and model-free reinforcement learning
Konovalov, A., & Krajbich, I. M. (2016) · Nature Communications · doi:10.1038/ncomms12438 · Cited by 86 · see citing works
Transfer of Learning Relates to Intrinsic Connectivity between Hippocampus, Ventromedial Prefrontal Cortex, and Large-Scale Networks
Gerraty, R. T., Davidow, J. Y., Wimmer, G. E., Kahn, I., & Daphna Shohamy, A. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.0185-14.2014 · Cited by 78 · see citing works
Adolescent Development of Value-Guided Goal Pursuit
Davidow, J. Y., Insel, C., & Somerville, L. H. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.05.003 · Cited by 81 · see citing works
Behavioral and Neural Signatures of Reduced Updating of Alternative Options in Alcohol-Dependent Patients during Flexible Decision-Making
Reiter, A. M. F., Deserno, L., Kallert, T. W., Heinze, H., Heinz, A., & Schlagenhauf, F. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.4322-15.2016 · Cited by 83 · see citing works
Decision-Theoretic Psychiatry
Huys, Q. J. M., Guitart‐Masip, M., Dolan, R. J., & Dayan, P. (2015) · Clinical Psychological Science · doi:10.1177/2167702614562040 · Cited by 81 · see citing works
Goal-oriented and habitual decisions: Neural signatures of model-based and model-free learning
Huang, Y., Yaple, Z. A., & Yu, R. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116834 · Cited by 79 · see citing works
Devaluation and sequential decisions: linking goal-directed and model-based behavior
Friedel, E., Koch, S., Wendt, J., Heinz, A., Deserno, L., & Schlagenhauf, F. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00587 · Cited by 79 · see citing works
Transition from ‘model-based’ to ‘model-free’ behavioral control in addiction: Involvement of the orbitofrontal cortex and dorsolateral striatum
Lucantonio, F., Caprioli, D., & Schoenbaum, G. M. (2013) · Neuropharmacology · doi:10.1016/j.neuropharm.2013.05.033 · Cited by 76 · see citing works
Posttraumatic Stress Disorder and the Developing Adolescent Brain
Cisler, J. M., & Herringa, R. J. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.06.001 · Cited by 81 · see citing works
Amygdala-cortical collaboration in reward learning and decision making
Wassum, K. M. (2022) · eLife · doi:10.7554/elife.80926 · Cited by 76 · see citing works
Emotions as ComputationsPreprint
Emanuel, A., & Eldar, E. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104977 · Cited by 83 · see citing works
Neurofeedback through the lens of reinforcement learning
Lubianiker, N., Paret, C., Dayan, P., & Hendler, T. (2022) · Trends in Neurosciences · doi:10.1016/j.tins.2022.03.008 · Cited by 73 · see citing works
Model-based learning: a synthesis of theory and research
Seel, N. M. (2017) · Educational Technology Research and Development · doi:10.1007/s11423-016-9507-9 · Cited by 78 · see citing works
Spatial Representations in the Human Brain
Herweg, N. A., & Kahana, M. J. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00297 · Cited by 70 · see citing works
Greater widespread functional connectivity of the caudate in older adults who practice kripalu yoga and vipassana meditation than in controls
Gard, T., Taquet, M., Dixit, R., Hölzel, B. K., Dickerson, B. C., & Lazar, S. W. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00137 · Cited by 65 · see citing works
Understanding psychiatric disorder by capturing ecologically relevant features of learning and decision-making
Scholl, J., & Klein-Flügge, M. C. (2017) · Behavioural Brain Research · doi:10.1016/j.bbr.2017.09.050 · Cited by 62 · see citing works
Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning
Kim, D., Park, G. Y., O’Doherty, J. P., & Lee, S. W. (2019) · Nature Communications · doi:10.1038/s41467-019-13632-1 · Cited by 64 · see citing works
Water optimization technique for precision irrigation system using IoT and machine learning
Vianny, D. M. M., John, A., Mohan, S., Sarlan, A. B., Adimoolam, M., & Ahmadian, A. (2022) · Sustainable Energy Technologies and Assessments · doi:10.1016/j.seta.2022.102307 · Cited by 63 · see citing works
Lateral prefrontal model-based signatures are reduced in healthy individuals with high trait impulsivity
Deserno, L., Wilbertz, T., Reiter, A. M. F., Horstmann, A., Neumann, J., Villringer, A., Heinze, H. –., & Schlagenhauf, F. (2015) · Translational Psychiatry · doi:10.1038/tp.2015.139 · Cited by 62 · see citing works
Issues or Identity? Cognitive Foundations of Voter Choice
Jenke, L., & Huettel, S. A. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.08.013 · Cited by 64 · see citing works
Computational models of drug use and addiction: A review.
Mollick, J., & Hedy Kober, A. (2020) · Journal of Abnormal Psychology · doi:10.1037/abn0000503 · Cited by 58 · see citing works
Explainable reinforcement learning for broad-XAI: a conceptual framework and survey
Dazeley, R., Vamplew, P. W., & Cruz, F. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08423-1 · Cited by 68 · see citing works
Safety out of control: dopamine and defence
Lloyd, K., & Dayan, P. (2016) · Behavioral and Brain Functions · doi:10.1186/s12993-016-0099-7 · Cited by 59 · see citing works
Striatal dopamine, reward, and decision making in schizophrenia
Deserno, L., Schlagenhauf, F., & Heinz, A. (2016) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2016.18.1/ldeserno · Cited by 58 · see citing works
A cortical circuit mechanism for structural knowledge-based flexible sensorimotor decision-making
Liu, Y., Xin, Y., & Xu, N.-L. (2021) · Neuron · doi:10.1016/j.neuron.2021.04.014 · Cited by 57 · see citing works
The neural basis of consciousness
Frith, C. (2019) · Psychological Medicine · doi:10.1017/s0033291719002204 · Cited by 57 · see citing works
Modeling other minds: Bayesian inference explains human choices in group decision-making
Khalvati, K., Park, S. A., Mirbagheri, S., Philippe, R., Sestito, M., Dreher, J., & Rao, R. P. N. (2019) · Science Advances · doi:10.1126/sciadv.aax8783 · Cited by 61 · see citing works
Reinforcement learning and its connections with neuroscience and psychology
Subramanian, A., Chitlangia, S., & Baths, V. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.10.003 · Cited by 57 · see citing works
Motivational Processes Underlying Substance Abuse Disorder
Meyer, P., King, C. P., & Ferrario, C. R. (2015) · Current topics in behavioral neurosciences · doi:10.1007/7854_2015_391 · Cited by 54 · see citing works
Slips of Action and Sequential Decisions: A Cross-Validation Study of Tasks Assessing Habitual and Goal-Directed Action Control
Sjoerds, Z., Dietrich, A., Deserno, L., de Wit, S., Villringer, A., Heinze, H., Schlagenhauf, F., & Horstmann, A. (2016) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2016.00234 · Cited by 52 · see citing works

What is dopamine doing in model-based reinforcement learning?Preprint
Akam, T. E., & Walton, M. E. (2020) · doi:10.31234/osf.io/z2fmw · Cited by 3 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166 · see citing works
Self-citation
Model-free and model-based reward prediction errors in EEG
Sambrook, T. D., Hardwick, B., Wills, A. J., & Goslin, J. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.05.023 · Cited by 51 · see citing works
Neuron Article States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning
Gläscher, J. P., Daw, N. D., Dayan, P., O’Doherty, J. P., Humanities, D. O., & Sciences, S. (2010)
Self-citation

Authors: Doll, B. B., Simon, D. A., & Daw, N. D.
Affiliations: Columbia University (US)
New York University (US)
Corresponding author: Daw, Nathaniel D
ORCID: Simon, Dylan A
Daw, Nathaniel D
Venue: Current Opinion in Neurobiology
Publisher: Elsevier BV
Published: 2012-12
Volume: 22
Issue: 6
Pages: 1075-1081
ISSN: 0959-4388 (ISSN-L)
DOI: 10.1016/j.conb.2012.08.003
PMC: PMC3513648
OpenAlex: W1977114987
Cited by: 380
References: 64
Citations by year: 2026: 13 · 2025: 26 · 2024: 40 · 2023: 26 · 2022: 35 · 2021: 34 · 2020: 33 · 2019: 29 · 2018: 29 · 2017: 17 · 2016: 32 · 2015: 31 · 2014: 21 · 2013: 13
FWCI: 1.65
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Licence: www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
Locations: Current Opinion in Neurobiology (publishedVersion)
PubMed (publishedVersion)
Europe PMC (PubMed Central) (acceptedVersion)
PubMed Central (acceptedVersion)
Funders: McKnight Foundation
McDonnell Foundation
NIMH (1R01MH087882-01)
NINDS (1R01NS078784-01)
Topics: Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords: model-based reinforcement learning, model-free reinforcement learning, reward prediction error, dopamine, decision-making, world models, neural correlates of learning, computational psychiatry
MeSH terms: Animals, Brain, Choice Behavior, Decision Making, Humans, Models, Neurological, Reinforcement, Psychology, Reward

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

The ubiquity of model-based reinforcement learning

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks