Model-Based Reasoning in Humans Becomes Automatic with Training
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Model-based and model-free reinforcement learning (RL) have been suggested as algorithmic realizations of goal-directed and habitual action strategies. Model-based RL is more flexible than model-free but requires sophisticated calculations using a learnt model of the world. This has led model-based RL to be identified with slow, deliberative processing, and model-free RL with fast, automatic processing. In support of this distinction, it has recently been shown that model-based reasoning is impaired by placing subjects under cognitive load--a hallmark of non-automaticity. Here, using the same task, we show that cognitive load does not impair model-based reasoning if subjects receive prior training on the task. This finding is replicated across two studies and a variety of analysis methods. Thus, task familiarity permits use of model-based reasoning in parallel with other cognitive demands. The ability to deploy model-based reasoning in an automatic, parallelizable fashion has widespread theoretical implications, particularly for the learning and execution of complex behaviors. It also suggests a range of important failure modes in psychiatric disorders.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Attention to Action: Willed and Automatic Control of Behavior
Norman, D. A., & Shallice, T. (1986) · Consciousness and Self-Regulation · doi:10.1007/978-1-4757-0629-1_1 · Cited by 4,455
Neural systems of reinforcement for drug addiction: from actions to habits to compulsion
Everitt, B. J., & Robbins, T. W. (2005) · Nature Neuroscience · doi:10.1038/nn1579 · Cited by 3,785 · cited 1×
“For example, over-reliance on habits could lead to inflexible decision-making in addiction [4] and compulsion [5].”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 5×
“We analyzed data using previously described reinforcement learning (RL) models [1,19], including a hybrid model and reduced (nested) versions that captured pure model-free and model-based choice.”
BackgroundMethodInfluentialLearning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Executive Function and the Frontal Lobes: A Meta-Analytic Review
Alvarez, J. A., & Emory, E. (2006) · Neuropsychology Review · doi:10.1007/s11065-006-9002-x · Cited by 1,981
Goal-directed instrumental action: contingency and incentive learning and their cortical substrates
Balleine, B. W., & Dickinson, A. (1998) · Neuropharmacology · doi:10.1016/s0028-3908(98)00033-1 · Cited by 1,541
A free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes
Badre, D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.004 · Cited by 980
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Self-citationHierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952 · cited 1×
“It is also consistent with the notion of "models" throughout processing hierarchies in the brain, from low-level sensory processing to high-level cognition [45,46].”
ResultOptimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948
Dorsolateral prefrontal contributions to human working memory
Barbey, A. K., Koenigs, M., & Grafman, J. (2013) · Cortex · doi:10.1016/j.cortex.2012.05.022 · Cited by 940
Neural correlates, computation and behavioural impact of decision confidence
Kepecs, A., Uchida, N., Zariwala, H. A., & Mainen, Z. F. (2008) · Nature · doi:10.1038/nature07200 · Cited by 809 · cited 1×
“More stochastic choice might reflect a reduction in decision confidence [39,40].”
BackgroundA specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · cited 1×
“For example, it is well established that extended training reduces sensitivity to outcome devaluation [13,41 – 44].”
BackgroundInstrumental Responding following Reinforcer Devaluation
Adams, C. D., & Dickinson, A. (1981) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748108400816 · Cited by 687 · cited 1×
“For example, it is well established that extended training reduces sensitivity to outcome devaluation [13,41 – 44].”
BackgroundDisorders of compulsivity: A common bias towards learning habits
Voon, V, Derbyshire, K., Rück, C., Irvine, M. A., Worbe, Y., Enander, J., Schreiber, L. R. N., Gillan, C., Fineberg, N. A., Sahakian, B. J., et al. (2015) · Molecular Psychiatry · doi:10.1038/mp.2014.44 · Cited by 643 · cited 2×
“For example, over-reliance on habits could lead to inflexible decision-making in addiction [4] and compulsion [5].”
BackgroundResultConfidence in value-based choice
De Martino, B., Fleming, S. M., Garrett, N., & Dolan, R. J. (2013) · Nature Neuroscience · doi:10.1038/nn.3279 · Cited by 534
Self-citationDetermining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529
Otto (2013) · doi:10.1177/0956797612463080 · Cited by 352 · cited 10×
“In keeping with other studies utilizing the two-step task [16,18,19,21], we repeated the regression analysis but now only considering the influence of events occurring on the immediately preceding trial.”
BackgroundMethodResultInfluentialSpeed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · cited 1×
“There may also be resource requirements for maintaining belief distributions over meta-parameters, such as whether the task structure changes or new fractals appear, what appropriate learning rates are, when modelbased reasoning should be deployed [1,30] and how attentional resources should be allocated within a trial.”
BackgroundThe effect of the instrumental training contingency on susceptibility to reinforcer devaluation
Dickinson, A., Nicholas, D. J., & Adams, C. D. (1983) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748308400912 · Cited by 338
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Self-citationDopamine enhances model-based over model-free choice behavior
Wunderlich, K., Smittenaar, P., & Dolan, R. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.042 · Cited by 294 · cited 1×
“Previous studies in our laboratory and others have shown that 20 to 25 participants provide sufficient power to quantify the contribution of model-free and model-based strategies in the two-step task [16,18,19,38].”
ResultSelf-citationOwen, A. M. (1997) · Progress in Neurobiology · doi:10.1016/s0301-0082(97)00042-7 · Cited by 251
Smittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.009 · Cited by 244 · cited 4×
“This is supported by observations that model-based reasoning is impaired under cognitive load [16] or acute stress [17], and following disruption of dorsolateral prefrontal cortex function via TMS [18], with the degree of impairment interacting with base-line working memory capacity.”
ResultInfluentialSelf-citationRetrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217 · cited 1×
“Indeed, a recent experiment has suggested that the model-based system can “train” the model-free system by replaying and simulating experience offline, and that this in turn allows for choice under load that appears model-based [36].”
BackgroundBeauchamp, M. H., Dagher, A., Aston, J. A. D., & Doyon, J. (2003) · NeuroImage · doi:10.1016/j.neuroimage.2003.07.003 · Cited by 148
Cited by
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
When Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211
“This task has proven to be a useful and popular tool to characterize the neural [8, 10 – 18], behavioral [19 – 31] and clinical [32 – 35] implications of dual-process models within the reinforcement learning framework.”
BackgroundMethodThe multiple effects of practice: skill, habit and reduced cognitive load
Haith, A. M., & Krakauer, J. W. (2018) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.01.015 · Cited by 206
The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection
Akam, T., Rodrigues-Vaz, I., Marcelo, I., Zhang, X., Pereira, M., Oliveira, R. F., Dayan, P., & Costa, R. M. (2021) · Neuron · doi:10.1016/j.neuron.2020.10.013 · Cited by 170
Metadata
- Authors
- Economides, M., Kurth-Nelson, Z., Lübbert, A., Guitart-Masip, M., & Dolan, R. J.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-09-17
- Volume
- 11
- Issue
- 9
- Pages
- e1004463
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2178583635
- Cited by
- 72
- References
- 28
- FWCI
- 3.32
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.