Neural computations underlying arbitration between model-based and model-free learning
Open accessAbstract
There is accumulating neural evidence to support the existence of two distinct systems for guiding action selection, a deliberative "model-based" and a reflexive "model-free" system. However, little is known about how the brain determines which of these systems controls behavior at one moment in time. We provide evidence for an arbitration mechanism that allocates the degree of control over behavior by model-based and model-free systems as a function of the reliability of their respective predictions. We show that the inferior lateral prefrontal and frontopolar cortex encode both reliability signals and the output of a comparison between those signals, implicating these regions in the arbitration process. Moreover, connectivity between these regions and model-free valuation areas is negatively modulated by the degree of model-based control in the arbitrator, suggesting that arbitration may work through modulation of the model-free valuation system when the arbitrator deems that the model-based system should drive behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Subsequent investigations favor a more dynamic integration in which both systems contribute to the computation of action values according to their relative reliabilities.
“However, subsequent investigations favor a more dynamic integration in which both systems contribute to the computation of action values according to their relative reliabilities (Lee et al. 2014).”
From Psychology of Habit. (2016) · cites this paper as Lee et al. 2014
Recent behavioural and neural evidence supports the idea that at any given moment action selection results from a mixture of goal-directed and habitual control, rather than being under the exclusive control of either system.
“Consistent with this view, recent behavioural and neural evidence supports the idea that at any given moment action selection results from a mixture of goal-directed and habitual control, rather than being under the exclusive control of either of the two systems (Daw et al., 2011, Lee et al., 2014, Otto et al., 2013).”
From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Lee et al., 2014
The results may also be consistent with models that combine contributions of different learning systems.
“Our model proposes one learning system with variable learning rates, but these results may also be consistent with models that combine contributions of different learning systems.73–75”
From Serotonin neurons modulate learning rate through uncertainty (2022) · cites this paper as [74]
The arbitrator selects between model-based and model-free systems according to criteria such as accuracy or prediction error or prevailing rate of reinforcement.
“Given the tradeoff, the arbitrator selects between the two systems according to various criteria, such as the accuracy or prediction error connected with each (Daw et al., 2005; Lee et al., 2014) or the prevailing rate of reinforcement (Dezfouli et al., 2014).”
From Context, attention, and the switch between habit and goal-direction in behavior (2021) · cites this paper as Lee et al., 2014
Reinforcement learning approaches have often accepted the coexistence of action and habit systems and allowed one or the other to be selected by a hypothetical arbitrator for behavioral control.
“Approaches to action and habit in the reinforcement learning tradition (e.g., Sutton & Barto, 1998) have also often accepted the coexistence of action and habit systems and allowed one or the other to be selected by a hypothetical “arbitrator” for behavioral control (e.g.,Daw et al., 2005 ; Lee et al., 2014).”
From Context, attention, and the switch between habit and goal-direction in behavior (2021) · cites this paper as Lee et al., 2014
If habits play a role in addiction, they may do so because the habit-action switch becomes sticky, making it difficult to switch back to goal-directed status once a habit is learned.
“If habits do play a role in addiction, they may do so because the habit-action switch becomes sticky (see also Balleine et al., 2009; Lee et al., 2014). As a consequence, once a habit is learned, it may be difficult to switch it back to goal-directed status.”
From Context, attention, and the switch between habit and goal-direction in behavior (2021) · cites this paper as Lee et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879 · see citing works
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Convergence Properties of the Nelder--Mead Simplex Method in Low Dimensions
Lagarias, J. C., Reeds, J. A., Wright, M. H., & Wright, P. E. (1998) · SIAM Journal on Optimization · doi:10.1137/s1052623496303470 · Cited by 7,449 · see citing works
(2009) · Cited by 6,153 · see citing works
ANew Look at the Statistical Model Identification
AKAlKE, H. (2011) · Cited by 4,177 · see citing works
Neural systems of reinforcement for drug addiction: from actions to habits to compulsion
Everitt, B. J., & Robbins, T. W. (2005) · Nature Neuroscience · doi:10.1038/nn1579 · Cited by 3,785 · see citing works
Inhibition and the right inferior frontal cortex
Aron, A. R., Robbins, T. W., & Poldrack, R. A. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.010 · Cited by 3,197 · see citing works
Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems
Dayan, P., & Abbott, L. F. (2001) · Cited by 3,153 · see citing works
Psychophysiological and Modulatory Interactions in Neuroimaging
Friston, K. J., Buechel, C., Fink, G. R., Morris, J., Rolls, E., & Dolan, R. J. (1997) · NeuroImage · doi:10.1006/nimg.1997.0291 · Cited by 3,097 · see citing works
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · see citing works
Individual Choice Behavior: A Theoretical Analysis.
Cane, V. R., & Luce, R. D. (1960) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2343282 · Cited by 2,598 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Barto, A. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,172 · see citing works
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · see citing works
Self-citationA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Zhang, B., Li, C., & Lin, N. (2020) · De Gruyter eBooks · doi:10.1515/9783110595567-011 · Cited by 1,939 · see citing works
Self-Control in Decision-Making Involves Modulation of the vmPFC Valuation System
Hare, T. A., Camerer, C. F., & Rangel, A. (2009) · Science · doi:10.1126/science.1168450 · Cited by 1,928 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775 · see citing works
Self-citationStop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans
Aron, A. R., Fletcher, P. C., Bullmore, E. T., Sahakian, B. J., & Robbins, T. W. (2003) · Nature Neuroscience · doi:10.1038/nn1003 · Cited by 1,771 · see citing works
Habits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · see citing works
Goal-directed instrumental action: contingency and incentive learning and their cortical substrates
Balleine, B. W., & Dickinson, A. (1998) · Neuropharmacology · doi:10.1016/s0028-3908(98)00033-1 · Cited by 1,541 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Self-citationBayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457 · see citing works
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423 · see citing works
Right hemispheric dominance of inhibitory control: An event-related functional MRI study
Garavan, H., Ross, T. J., & Stein, E. A. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.14.8301 · Cited by 1,394 · see citing works
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226 · see citing works
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166 · see citing works
Self-citationIndividual choice behavior: A theoretical analysis.
Luce, R. D. (2005) · doi:10.1037/14396-000 · Cited by 1,157 · cited 2× · see citing works
“Finally, the arbitration model selects actions stochastically according to the following softmax function (Gläscher et al., 2010; Luce, 1959) (the”
MethodFrontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111 · see citing works
The role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099 · see citing works
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · see citing works
Scientific Charge-Coupled Devices
Janesick, J. R., Elliott, T. S., Collins, S. A., et al. (1987) · Optical Engineering · doi:10.1117/12.7974139 · Cited by 863 · see citing works
Human insula activation reflects risk prediction errors as well as risk
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4286-07.2008 · Cited by 784 · see citing works
Cools, R., Clark, L., Owen, A. M., & Robbins, T. W. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-11-04563.2002 · Cited by 784 · see citing works
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · see citing works
Self-citationAnterior Prefrontal Function and the Limits of Human Decision-Making
Koechlin, É., & Hyafil, A. (2007) · Science · doi:10.1126/science.1142995 · Cited by 750 · see citing works
Instrumental Responding following Reinforcer Devaluation
Adams, C. D., & Dickinson, A. (1981) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748108400816 · Cited by 687 · see citing works
Boorman, E. D., Behrens, T. E. J., Woolrich, M. W., & Rushworth, M. F. S. (2009) · Neuron · doi:10.1016/j.neuron.2009.05.014 · Cited by 665 · see citing works
Probabilistic population codes for Bayesian decision making
Beck, J. M., Ma, W. J., Kiani, R., Hanks, T., Churchland, A. K., Roitman, J., Shadlen, M. N., Latham, P. E., & Pouget, A. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.021 · Cited by 645 · see citing works
Chib, V. S., Rangel, A., Shimojo, S., & O'Doherty, J. P. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2575-09.2009 · Cited by 643 · see citing works
Self-citationGillan, C. M., Papmeyer, M., Morein-Zamir, S., Sahakian, B. J., Fineberg, N. A., Robbins, T. W., & de Wit, S. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.10071062 · Cited by 609 · see citing works
Confidence in value-based choice
De Martino, B., Fleming, S. M., Garrett, N., & Dolan, R. J. (2013) · Nature Neuroscience · doi:10.1038/nn.3279 · Cited by 534 · see citing works
Determining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529 · see citing works
Self-citationHampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498 · see citing works
Self-citationRole of the lateral prefrontal cortex in executive behavioral control.
Tanji, J., & Hoshi, E. (2008) · Physiological Reviews · doi:10.1152/physrev.00014.2007 · Cited by 495 · see citing works
Multiple model-based reinforcement learning
Doya, K., Samejima, K., Katagiri, K.-I., & Kawato, M. (2002) · Neural Computation · doi:10.1162/089976602753712972 · Cited by 476 · see citing works
Optogenetic stimulation of lateral orbitofronto-striatal pathway suppresses compulsive behaviors
Burguière, E., Monteiro, P., Feng, G., & Graybiel, A. M. (2013) · Science · doi:10.1126/science.1232380 · Cited by 457 · see citing works
de Wit, S., Watson, P., Harsay, H. A., Cohen, M. X., van de Vijver, I., & Ridderinkhof, K. R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.1088-12.2012 · Cited by 373 · see citing works
Transformation of stimulus value signals into motor commands during simple choice
Hare, T. A., Schultz, W., Camerer, C. F., O’Doherty, J. P., & Rangel, A. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1109322108 · Cited by 368 · see citing works
Self-citationDifferential roles of human striatum and amygdala in associative learning
Li, J., Schiller, D., Schoenbaum, G. M., Phelps, E. A., & Daw, N. D. (2011) · Nature Neuroscience · doi:10.1038/nn.2904 · Cited by 364 · see citing works
Le Pelley, M. E. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000141 · Cited by 360 · see citing works
Coutureau, E., & Killcross, S. (2003) · Behavioural Brain Research · doi:10.1016/j.bbr.2003.09.025 · Cited by 342 · see citing works
Lim, S.-L., O'Doherty, J. P., & Rangel, A. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.1246-11.2011 · Cited by 329 · see citing works
Self-citationBaird, B., Smallwood, J., Gorgolewski, K. J., & Margulies, D. S. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.0786-13.2013 · Cited by 315 · see citing works
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310 · see citing works
Rostrolateral Prefrontal Cortex and Individual Differences in Uncertainty-Driven Exploration
Badre, D., Doll, B. B., Long, N. M., & Frank, M. J. (2012) · Neuron · doi:10.1016/j.neuron.2011.12.025 · Cited by 309 · see citing works
Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280 · see citing works
Calculating Consequences: Brain Systems That Encode the Causal Effects of Actions
Tanaka, S. C., Balleine, B. W., & O'Doherty, J. P. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1808-08.2008 · Cited by 268 · see citing works
Self-citationRisk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings
Payzan-LeNestour, E., & Bossaerts, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001048 · Cited by 267 · see citing works
Avoiding non-independence in fMRI data analysis: Leave one subject out
Esterman, M. S., Tamber-Rosenau, B. J., Chiu, Y., & Yantis, S. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.10.092 · Cited by 262 · see citing works
Contributions of Striatal Subregions to Place and Response Learning
Yin, H. H., & Knowlton, B. J. (2004) · Learning & Memory · doi:10.1101/lm.81004 · Cited by 238 · see citing works
Krugel, L. K., Biele, G., Mohr, P. N. C., Li, S.-C., & Heekeren, H. R. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0905191106 · Cited by 229 · see citing works
Contributions of the ventromedial prefrontal cortex to goal-directed action selection
O’Doherty, J. P. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06290.x · Cited by 225 · see citing works
de Wit, S., Corlett, P. R., Aitken, M. R., Dickinson, A., & Fletcher, P. C. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1639-09.2009 · Cited by 200 · see citing works
Smith, K. S., Virkud, A., Deisseroth, K., & Graybiel, A. M. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216264109 · Cited by 173 · see citing works
Adapting bias by gradient descent: an incremental version of delta-bar-delta
Sutton, R. S. (1992) · Cited by 169 · see citing works
Neural Correlates of Variations in Event Processing during Learning in Basolateral Amygdala
Roesch, M. R., Calu, D. J., Esber, G. R., & Schoenbaum, G. M. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.5781-09.2010 · Cited by 150 · see citing works
Rowe, J. B., Hughes, L. E., & Nimmo‐Smith, I. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.02.045 · Cited by 72 · see citing works
Neural Substrates for Reversing Stimulus–Outcome and Stimulus–Response Associations
Xue, G., Ghahremani, D. G., & Poldrack, R. A. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.4001-08.2008 · Cited by 47 · see citing works
Du, K.-L., & Swamy, M. N. S. (2019) · Springer eBooks · doi:10.1007/978-1-4471-7452-3_17 · Cited by 46 · see citing works
Separate encoding of model-based and model-free valuations in the human brain
Beierholm, U., Anen, C., Quartz, S. R., & Bossaerts, P. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.071 · Cited by 41 · see citing works
How initial confirmatory experience potentiates the detrimental influence of bad advice
Staudinger, M. R., & Büchel, C. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.02.074 · Cited by 33 · see citing works
Pennini, F., & Plastino, A. (2010) · Journal of Physics Conference Series · doi:10.1088/1742-6596/246/1/012030 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445 · see citing works
The neural bases of emotion regulation
Etkin, A., Büchel, C., & Gross, J. J. (2015) · Nature Reviews Neuroscience · doi:10.1038/nrn4044 · Cited by 1,244 · see citing works
Emotion and the prefrontal cortex: an integrative review
Dixon, M. L., Thiruchselvam, R., Todd, R., & Christoff, K. (2017) · Psychological Bulletin · doi:10.1037/bul0000096 · Cited by 745 · see citing works
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., O⿿Doherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674 · see citing works
Self-citationActive Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · see citing works
“…belief precisions or related constructs to modulate the balance of model based and model free controllers (Daw et al., 2005; Gershman and Daw, 2012; Lee et al., 2014; Pezzulo et al., 2013), there are some unique features of the Active Inference scheme that make novel and empirically testable…”
InfluentialLearning, reward, and decision making
O'Doherty, J. P., Cockburn, J., & Pauli, W. M. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-010416-044216 · Cited by 475 · see citing works
Self-citationReinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Goal-directed learning and obsessive-compulsive disorder
Gillan, C. M., & Robbins, T. W. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0475 · Cited by 348 · see citing works
The role of habit in compulsivity
Gillan, C. M., Robbins, T. W., Sahakian, B. J., van den Heuvel, O. A., & van Wingen, G. (2016) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2015.12.033 · Cited by 292 · see citing works
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · see citing works
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280 · see citing works
Cognitive inflexibility in Obsessive-Compulsive Disorder
Gruner, P., & Pittenger, C. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2016.07.030 · Cited by 261 · see citing works
When Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211 · see citing works
Deserno, L., Huys, Q. J. M., Boehme, R., Buchert, R., Heinze, H.-J., Grace, A. A., Dolan, R. J., Heinz, A., & Schlagenhauf, F. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1417219112 · Cited by 233 · see citing works
Barch, D. M., Pagliaccio, D., & Luking, K. (2015) · Current Topics in Behavioral Neurosciences · doi:10.1007/7854_2015_376 · Cited by 218 · see citing works
Serotonin neurons modulate learning rate through uncertainty
Grossman, C. D., Bari, B. A., & Cohen, J. Y. (2022) · Current Biology · doi:10.1016/j.cub.2021.12.006 · Cited by 164 · see citing works
The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection
Akam, T., Rodrigues-Vaz, I., Marcelo, I., Zhang, X., Pereira, M., Oliveira, R. F., Dayan, P., & Costa, R. M. (2021) · Neuron · doi:10.1016/j.neuron.2020.10.013 · Cited by 170 · see citing works
Meta-learning in natural and artificial intelligence
Wang, J. X. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.01.002 · Cited by 171 · see citing works
Enhanced habit formation in Gilles de la Tourette syndrome
Delorme, C., Salvador, A., Valabrègue, R., Roze, E., Palminteri, S., Vidailhet, M., de Wit, S., Robbins, T., Hartmann, A., & Worbe, Y. (2016) · Brain · doi:10.1093/brain/awv307 · Cited by 154 · see citing works
Habits as action sequences: hierarchical action control and changes in outcome value
Dezfouli, A., Lingawi, N. W., & Balleine, B. W. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0482 · Cited by 139 · see citing works
Dopamine, reward learning, and active inference
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00136 · Cited by 112 · see citing works
Model averaging, optimal inference, and habit formation
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. J. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00457 · Cited by 121 · see citing works
Iigaya, K., Fonseca, M. S., Murakami, M., Mainen, Z. F., & Dayan, P. (2018) · Nature Communications · doi:10.1038/s41467-018-04840-2 · Cited by 120 · see citing works
Incentives Boost Model-Based Control Across a Range of Severity on Several Psychiatric Constructs
Patzelt, E. H., Kool, W., Millner, A. J., & Gershman, S. J. (2019) · Biological Psychiatry · doi:10.1016/j.biopsych.2018.06.018 · Cited by 94 · see citing works
Reduced model-based decision-making in schizophrenia.
Culbreth, A. J., Westbrook, A., Daw, N. D., Botvinick, M., & Barch, D. M. (2016) · Journal of Abnormal Psychology · doi:10.1037/abn0000164 · Cited by 104 · see citing works
Context, attention, and the switch between habit and goal-direction in behavior
Bouton, M. E. (2021) · Learning & Behavior · doi:10.3758/s13420-021-00488-z · Cited by 100 · see citing works
An insula hierarchical network architecture for active interoceptive inference
Fermin, A. S. R., Friston, K., & Yamawaki, S. (2022) · Royal Society Open Science · doi:10.1098/rsos.220226 · Cited by 101 · see citing works
Vinckier, F., Gaillard, R., Palminteri, S., Rigoux, L., Salvador, A., Fornito, A., Adapa, R., Krebs, M. O., Pessiglione, M., & Fletcher, P. C. (2016) · Molecular Psychiatry · doi:10.1038/mp.2015.73 · Cited by 87 · see citing works
Scaling prediction errors to reward variability benefits error-driven learning in humans
Diederen, K. M. J., & Schultz, W. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00483.2015 · Cited by 69 · see citing works
Conflicted between goal-directed and habitual control, an fMRI investigation
Watson, P., van Wingen, G., & de Wit, S. (2018) · eneuro · doi:10.1523/eneuro.0240-18.2018 · Cited by 84 · see citing works
Stress-sensitive inference of task controllability
Ligneul, R., Mainen, Z. F., Ly, V., & Cools, R. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01306-w · Cited by 60 · see citing works
Arbitration between action strategies in obsessive-compulsive disorder
Gruner, P., Anticevic, A., Lee, D., & Pittenger, C. (2016) · The Neuroscientist · doi:10.1177/1073858414568317 · Cited by 62 · see citing works
Dynamic Neural Interactions Supporting the Cognitive Reappraisal of Emotion
Steward, T., Davey, C. G., Jamieson, A. J., Stephanou, K., Soriano-Mas, C., Felmingham, K. L., & Harrison, B. J. (2021) · Cerebral Cortex · doi:10.1093/cercor/bhaa268 · Cited by 52 · see citing works
Epistemic Communities under Active Inference
Albarracin, M., Demekas, D., Ramstead, M. J. D., & Heins, C. (2022) · Entropy · doi:10.3390/e24040476 · Cited by 48 · see citing works
“145 If behavior is initially goal- or information-driven, habit-learning can then ‘zero-in’ and 146 isolate the invariant features of such (initially) goal-directed behavior [75], mirroring the 147 so-called transition from ‘model-based‘ to ‘model-free‘ decision-making in reinforcement 148 learning [76,77].”
BackgroundMethodMiyamoto, K., Trudel, N., Kamermans, K., Lim, M. C., Lazari, A., Verhagen, L., Wittmann, M. K., & Rushworth, M. F. S. (2021) · Neuron · doi:10.1016/j.neuron.2021.02.024 · Cited by 45 · see citing works
Sebold, M., Schad, D. J., Nebe, S., Garbusow, M., Jünger, E., Kroemer, N. B., Kathmann, N., Zimmermann, U. S., Smolka, M. N., Rapp, M. A., et al. (2016) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00945 · Cited by 41 · see citing works
Corticostriatal foundations of habits
Seger, C. A. (2018) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.01.006 · Cited by 38 · see citing works
Recent developments in the habit hypothesis of OCD and compulsive disorders
Gillan, C. M. (2021) · Current Topics in Behavioral Neurosciences · doi:10.1007/7854_2020_199 · Cited by 27 · see citing works
Imagining the future self through thought experiments
Miyamoto, K., Rushworth, M. F. S., & Shea, N. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.01.005 · Cited by 28 · see citing works
Stressful Events as Teaching Signals for the Brain
Trapp, S., O’Doherty, J. P., & Schwabe, L. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.03.007 · Cited by 25 · see citing works
Self-citationColas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
Self-citation
- Authors
- Lee, S. W., Shimojo, S., & O’Doherty, J. P.
- Affiliations
- California Institute of Technology (US)
- Corresponding author
- Lee, Sang Wan
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2014-02
- Volume
- 81
- Issue
- 3
- Pages
- 687-699
- ISSN
- 0896-6273 (ISSN-L)
- PMC
- PMC3968946
- OpenAlex
- W2129299900
- Cited by
- 642
- References
- 77
- Citations by year
- 2026: 41 · 2025: 43 · 2024: 70 · 2023: 50 · 2022: 48 · 2021: 67 · 2020: 61 · 2019: 58 · 2018: 45 · 2017: 51 · 2016: 46 · 2015: 47 · 2014: 14
- FWCI
- 29.37
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Neuron (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- Funders
- NIH (DA033077-01)
- OppNet
- Gordon and Betty foundation
- JST-CREST
- Caltech-Tamagawa gCOE
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- model-based learning, model-free learning, arbitration mechanism, forecast reliability, action selection, frontopolar cortex, neural control of behavior
- MeSH terms
- Adult, Brain, Brain Mapping, Computer Simulation, Decision Making, Female, Goals, Humans, Image Processing, Computer-Assisted, Learning, Magnetic Resonance Imaging, Male, Models, Neurological, Oxygen, Psychophysics, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
