Model-free decision making resists improved instructions and is enhanced by stimulus-response associations
Open accessPublished version of a preprint
Abstract
Human behaviour may be thought of as supported by two different computational-learning mechanisms, model-free and model-based respectively. In model-free strategies, stimulus-response associations are strengthened when actions are followed by a reward and weakened otherwise. In model-based learning, previous to selecting an action, the current values of the different possible actions are computed based on a detailed model of the environment. Previous research with the two-stage task suggests that participants' behaviour usually shows a mixture of both strategies. But, interestingly, a recent study by da Silva and Hare (2020) found that participants primarily deploy model-based behaviour when they are given detailed instructions about the structure of the task. In the present study, we reproduce this essential experiment. Our results confirm that improved instructions give rise to a stronger model-based component. Crucially, we also found a significant effect of reward that became stronger under conditions that favoured the development of strong stimulus-response associations. This suggests that the effect of reward, often taken as indicator of a model-free component, is related to stimulus-response learning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,308 · see citing works
The VideoToolbox software for visual psychophysics: transforming numbers into movies
Pelli, D. G. (1997) · Spatial Vision · doi:10.1163/156856897x00366 · Cited by 11,110 · see citing works
<i>Stan</i> : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · see citing works
Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.03.013 · Cited by 4,147 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
How are habits formed: modelling habit formation in the real world
Lally, P., van Jaarsveld, C. H. M., Potts, H. W. W., & Wardle, J. (2010) · European Journal of Social Psychology · doi:10.1002/ejsp.674 · Cited by 2,036 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217 · see citing works
Gardner, B. (2015) · Health Psychology Review · doi:10.1080/17437199.2013.876238 · Cited by 1,016 · see citing works
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996 · see citing works
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
The Pull of the Past: When Do Habits Persist Despite Conflict With Motives?
Neal, D. T., Wood, W., Wu, M., & Kurlander, D. (2011) · Personality and Social Psychology Bulletin · doi:10.1177/0146167211419863 · Cited by 298 · see citing works
Motivational control after extended instrumental training
Dickinson, A., Balleine, B. W., Watt, A. A., González, F., & Boakes, R. A. (1995) · Animal Learning & Behavior · doi:10.3758/bf03199935 · Cited by 286 · see citing works
Exercise habit formation in new gym members: a longitudinal study
Kaushal, N., & Rhodes, R. E. (2015) · Journal of Behavioral Medicine · doi:10.1007/s10865-015-9640-7 · Cited by 285 · see citing works
Model-based learning protects against forming habits
Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0347-6 · Cited by 283 · see citing works
From Creatures of Habit to Goal-Directed Learners
Decker, J. H., Otto, A. R., Daw, N. D., & Hartley, C. A. (2016) · Psychological Science · doi:10.1177/0956797616639301 · Cited by 265 · see citing works
Dorsal hippocampus contributes to model-based planning
Miller, K. J., Botvinick, M., & Brody, C. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4613 · Cited by 243 · see citing works
Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems
Kool, W., Gershman, S. J., & Cushman, F. A. (2017) · Psychological Science · doi:10.1177/0956797617708288 · Cited by 240 · see citing works
Shifting the balance between goals and habits: Five failures in experimental habit induction.
de Wit, S., Kindt, M., Knot, S. L., Verhoeven, A. A. C., Robbins, T. W., Gasull-Camós, J., Evans, M., Mirza, H., & Gillan, C. M. (2018) · Journal of Experimental Psychology General · doi:10.1037/xge0000402 · Cited by 217 · see citing works
Habitual Behavior Is Goal-Driven
Kruglanski, A. W., & Szumowska, E. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620917676 · Cited by 217 · see citing works
When Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211 · see citing works
Time-dependent competition between goal-directed and habitual response preparation
Hardwick, R. M., Forrence, A. D., Krakauer, J. W., & Haith, A. M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0725-0 · Cited by 211 · see citing works
The Power of Goal-Directed Processes in the Causation of Emotional and Other Actions
Moors, A., Boddez, Y., & Houwer, J. D. (2017) · Emotion Review · doi:10.1177/1754073916669595 · Cited by 180 · see citing works
Humans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153 · see citing works
The relationship between habits and motor skills in humans
Du, Y., Krakauer, J. W., & Haith, A. M. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.02.002 · Cited by 94 · see citing works
Kicking the habit: Why evidence for habits in humans might be overestimated.
Houwer, J. D., Tanaka, A., Moors, A., & Tibboel, H. (2017) · Motivation Science · doi:10.1037/mot0000065 · Cited by 89 · see citing works
Measuring habit formation through goal-directed response switching.
Luque, D., Molinero, S., Watson, P., López, F. J., & Pelley, M. E. L. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000722 · Cited by 75 · see citing works
Self-citationModel-Based Reasoning in Humans Becomes Automatic with Training
Economides, M., Kurth-Nelson, Z. L., Lübbert, A., Guitart‐Masip, M., & Dolan, R. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004463 · Cited by 72 · see citing works
Planning Complexity Registers as a Cost in Metacontrol
Kool, W., Gershman, S. J., & Cushman, F. A. (2018) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01263 · Cited by 57 · see citing works
A goal-directed account of action slips: The reliance on old contingencies.
Buabang, E. K., Köster, M., Boddez, Y., Dessel, P. V., Houwer, J. D., & Moors, A. (2022) · Journal of Experimental Psychology General · doi:10.1037/xge0001280 · Cited by 38 · see citing works
Moors, A., Fini, C., Everaert, T., Bardi, L., Bossuyt, E., Kuppens, P., & Braß, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0217266 · Cited by 34 · see citing works
Rethinking model-based and model-free influences on mental effort and striatal prediction errors
da Silva, C. F., Lombardi, G., Edelson, M. G., & Hare, T. A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01573-1 · Cited by 31 · see citing works
Control becomes habitual early on when learning a novel motor skill
Yang, C. S., Cowan, N. J., & Haith, A. M. (2022) · Journal of Neurophysiology · doi:10.1152/jn.00273.2022 · Cited by 16 · see citing works
Effects of response delays and of unknown stimulus‐response mappings on the oddball effect on P3
Verleger, R., Grauhan, N., & Śmigasiewicz, K. (2016) · Psychophysiology · doi:10.1111/psyp.12756 · Cited by 14 · see citing works
Molinero, S., Giménez‐Fernández, T., López, F. J., Carretié, L., & Luque, D. (2021) · Psychophysiology · doi:10.1111/psyp.13795 · Cited by 8 · see citing works
Self-citationThe oddball effect on P3 disappears when feature relevance or feature-response mappings are unknown
Verleger, R., Keppeler, M., Sassenhagen, J., & Śmigasiewicz, K. (2018) · Experimental Brain Research · doi:10.1007/s00221-018-5334-z · Cited by 7 · see citing works
The relationship between habits and motor skills in humansPreprint
Du, Y., Krakauer, J. W., & Haith, A. M. (2021) · doi:10.31234/osf.io/9qrgd · Cited by 5 · see citing works
A new take on model-based and model-free influences on mental effort and striatal prediction errors
da Silva, C. F., Lombardi, G., Edelson, M. G., & Hare, T. A. (2023) · Conference on Cognitive Computational Neuroscience (CCN) · doi:10.32470/ccn.2023.1107-0 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015). Modelbased learning protects against forming habits. Cognitive, Affective, & Behavioural Neuroscience, 15 (3), 523 e 536. http://refhub.elsevier.com/S0010-9452(2300168-5/sref14)
- Kleiner, M., Brainard, D. H., & Pelli, D. G. (2007). What ' s new in Psychtoolbox-3? Perception, 36 (ECVP Abstract Supplement). http://refhub.elsevier.com/S0010-9452(2300168-5/sref18)
- Stan Development Team. (2012). Stan modeling language user ' s guide and reference manual Version 1.0. . http://refhub.elsevier.com/S0010-9452(2300168-5/sref34)
Cited by
Sorted by citations
Leveraging cognitive neuroscience for making and breaking real-world habits
Buabang, E. K., Donegan, K. R., Rafei, P., & Gillan, C. M. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.10.006 · Cited by 34 · see citing works
Knolle, F., Sen, P., Culbreth, A., Koch, K., Schmitz-Koep, B., Gürsel, D. A., Wunderlich, K., Avram, M., Berberich, G., Sorg, C., et al. (2024) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.240020 · Cited by 9 · see citing works
Held, L. K., Lesage, E., Kool, W., & Braem, S. (2025) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-025-01350-9 · Cited by 2 · see citing works
- Authors
- Luna, R., Vadillo, M. A., & Luque, D.
- Affiliations
- Consejo Superior de Investigaciones Científicas (ES)
- Universidad Autónoma de Madrid (ES)
- Corresponding author
- Luna, Raúl
- Luque, David
- Venue
- Cortex
- Publisher
- Elsevier BV
- Published
- 2023-11
- Volume
- 168
- Pages
- 102-113
- ISSN
- 0010-9452 (ISSN-L)
- OpenAlex
- W4384818504
- Cited by
- 3
- References
- 39
- Citations by year
- 2025: 1 · 2024: 2
- FWCI
- 0.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Cortex (publishedVersion, cc-by-nc-nd)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, cc-by)
- DIGITAL.CSIC (Spanish National Research Council (CSIC)) (submittedVersion, cc-by-nc-nd)
- Hispana (submittedVersion)
- Funders
- Gobierno de España Ministerio de Ciencia e Innovación
- Junta de Andalucía
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- model-free learning, model-based learning, decision-making, stimulus-response associations, reward, task instructions, reinforcement learning
- MeSH terms
- Adult, Decision Making, Female, Humans, Learning, Male, Psychomotor Performance, Reaction Time, Reward, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
