The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Cognitive Control Predicts Use of Model-based Reinforcement Learning
Open accessAbstract
Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is a matter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information—in the service of overcoming habitual, stimulus-driven responses—in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions between model-based and model-free choice behavior.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Mixed-Effects Models in S and S-PLUS
(2001) · Technometrics · doi:10.1198/tech.2001.s574 · Cited by 9,278
Neural systems of reinforcement for drug addiction: from actions to habits to compulsion
Everitt, B. J., & Robbins, T. W. (2005) · Nature Neuroscience · doi:10.1038/nn1579 · Cited by 3,785 · cited 2×
“…of substance abuse ascribe compulsive and drug-seeking behaviors to the aberrant expression of habitual or stimulus-driven control systems at the expense of goal-directed action (Everitt & Robbins, 2005), respectively instantiated in dual-system RL as the model-free and model-based systems.”
BackgroundConflict monitoring and anterior cingulate cortex: an update
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.10.003 · Cited by 3,623
Anterior Cingulate Conflict Monitoring and Adjustments in Control
Kerns, J. G., Cohen, J. D., MacDonald, A. W., Cho, R. Y., Stenger, V. A., & Carter, C. S. (2004) · Science · doi:10.1126/science.1089910 · Cited by 2,891 · cited 2×
“The study was approved by the University Committee on Activities Involving Human Subjects of New York University.”
MethodOut of Control: Visceral Influences on Behavior
Loewenstein, G. (1996) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1996.0028 · Cited by 2,610 · cited 1×
“A number of theories across neuroscience, cognitive psychology, and economics posit that choices may arise from at least two distinct systems (Dolan & Dayan, 2013; Kahneman, 2011; Balleine & O ʼ Doherty, 2009; Daw, Niv, & Dayan, 2005; Loewenstein, 1996).”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationThe variable nature of cognitive control: a dual mechanisms framework
Braver, T. S. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.12.010 · Cited by 2,482
Self-Control in Decision-Making Involves Modulation of the vmPFC Valuation System
Hare, T. A., Camerer, C. F., & Rangel, A. (2009) · Science · doi:10.1126/science.1168450 · Cited by 1,928
Cited by
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445
Learning-induced autonomy of sensorimotor systems
Bassett, D. S., Yang, M., Wymbs, N. F., & Grafton, S. T. (2015) · Nature Neuroscience · doi:10.1038/nn.3993 · Cited by 603
Cognitive effort: A neuroeconomic approach
Westbrook, A., & Braver, T. S. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0334-y · Cited by 606
Kool, W., & Botvinick, M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0401-9 · Cited by 288
The role of habit in compulsivity
Gillan, C. M., Robbins, T. W., Sahakian, B. J., van den Heuvel, O. A., & van Wingen, G. (2016) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2015.12.033 · Cited by 292
Model-based learning protects against forming habits
Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0347-6 · Cited by 283
Self-citationMiller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280
Decker, J. H., Otto, A. R., Daw, N. D., & Hartley, C. A. (2016) · Psychological Science · doi:10.1177/0956797616639301 · Cited by 263
Self-citationBarch, D. M., Pagliaccio, D., & Luking, K. (2015) · Current Topics in Behavioral Neurosciences · doi:10.1007/7854_2015_376 · Cited by 218
When Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211
“Furthermore, the degree of model-free control on this task is predicted by individual difference measures such as working memory capacity [23], cognitive control ability [24], processing speed [29], age [20, 31], extraversion [30], and even psychiatric pathology [11, 33, 34].”
BackgroundCognitive Control: Core Constructs and Current Considerations
Cohen, J. D. (2017) · The Wiley Handbook of Cognitive Control · doi:10.1002/9781118920497.ch1 · Cited by 179
Humans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153
The opportunity cost of time modulates cognitive effort
Otto, A. R., & Daw, N. D. (2019) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.05.006 · Cited by 138
Self-citationReasoning supports utilitarian resolutions to moral dilemmas across diverse measures.
Patil, I., Zucchelli, M. M., Kool, W., Campbell, S., Fornasier, F., Calò, M., Silani, G., Cikara, M., & Cushman, F. (2021) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000281 · Cited by 133
The global workspace needs metacognition
Shea, N., & Frith, C. D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.04.007 · Cited by 130
“Similarly, model-based reasoning, which calls for the manipulation of probabilities in working memory, is stronger in subjects who exercise more cognitive control in standard tasks [67] and is impaired by cognitive load [68].”
BackgroundReduced model-based decision-making in schizophrenia.
Culbreth, A. J., Westbrook, A., Daw, N. D., Botvinick, M., & Barch, D. M. (2016) · Journal of Abnormal Psychology · doi:10.1037/abn0000164 · Cited by 104
Self-citationSandra, D. A., & Otto, A. R. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.004 · Cited by 107
Self-citationLearning to lose control: A process-based account of behavioral addiction
Perales, J. C., King, D. L., Navas, J. F., Schimmenti, A., Sescousse, G., Starcevic, V., van Holst, R. J., & Billieux, J. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.12.025 · Cited by 101
Applying novel technologies and methods to inform the ontology of self-regulation
Eisenberg, I. W., Bissett, P. G., Canning, J. R., Dallery, J., Enkavi, A. Z., Whitfield-Gabrieli, S., Gonzalez, O., Green, A. I., Greene, M. A., Kiernan, M., et al. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.014 · Cited by 61
Lloyd, A., McKay, R., Sebastian, C. L., & Balsters, J. H. (2021) · Developmental Science · doi:10.1111/desc.13075 · Cited by 49
Sebold, M., Schad, D. J., Nebe, S., Garbusow, M., Jünger, E., Kroemer, N. B., Kathmann, N., Zimmermann, U. S., Smolka, M. N., Rapp, M. A., et al. (2016) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00945 · Cited by 41
Computational approaches to schizophrenia: A perspective on negative symptoms
Deserno, L., Heinz, A., & Schlagenhauf, F. (2017) · Schizophrenia Research · doi:10.1016/j.schres.2016.10.004 · Cited by 31
Model-based learning and individual differences in depression: The moderating role of stress
Heller, A. S., Ezie, C. E. C., Otto, A. R., & Timpano, K. R. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.09.007 · Cited by 29
Self-citationRecent developments in the habit hypothesis of OCD and compulsive disorders
Gillan, C. M. (2021) · Current Topics in Behavioral Neurosciences · doi:10.1007/7854_2020_199 · Cited by 27
Using computational models of learning to advance cognitive behavioral therapy
Berwian, I. M., Hitchock, P., Pisupati, S., Schoen, G., & Niv, Y. (2025) · Communications Psychology · doi:10.1038/s44271-025-00251-4 · Cited by 16
The neurobehavioral mechanisms of motivational control in attention-deficit/hyperactivity disorder
Ceceli, A. O., Natsheh, J. Y., Cruz, D., & Tricomi, E. (2020) · Cortex · doi:10.1016/j.cortex.2020.02.009 · Cited by 9
Knolle, F., Sen, P., Culbreth, A., Koch, K., Schmitz-Koep, B., Gürsel, D. A., Wunderlich, K., Avram, M., Berberich, G., Sorg, C., et al. (2024) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.240020 · Cited by 9
Metadata
- Authors
- Otto, A. R., Skatova, A., Madlon-Kay, S., & Daw, N. D.
- Venue
- Journal of Cognitive Neuroscience
- Publisher
- MIT Press - Journals
- Published
- 2015-02-01
- Volume
- 27
- Issue
- 2
- Pages
- 319-333
- ISSN
- 0898-929X (ISSN-L)
- 1530-8898
- OpenAlex
- W2119328422
- Cited by
- 186
- References
- 34
- FWCI
- 9.87
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.