The DOI matches its Crossref record; 65 of the 68 references it cites were found in a registry or an index, and 3 are not checked yet.
Humans primarily use model-based inference in the two-stage task
Open accessPublished version of a preprint
Abstract
Distinct model-free and model-based learning processes are thought to drive both typical and dysfunctional behaviours. Data from two-stage decision tasks have seemingly shown that human behaviour is driven by both processes operating in parallel. However, in this study, we show that more detailed task instructions lead participants to make primarily model-based choices that have little, if any, simple model-free influence. We also demonstrate that behaviour in the two-stage task may falsely appear to be driven by a combination of simple model-free and model-based learning if purely model-based agents form inaccurate models of the task because of misconceptions. Furthermore, we report evidence that many participants do misconceive the task in important ways. Overall, we argue that humans formulate a wide variety of learning models. Consequently, the simple dichotomy of model-free versus model-based learning is inadequate to explain behaviour in the two-stage task and connections between reward learning, habit formation and compulsivity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,968
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
A framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Visual fixations and the computation and comparison of value in simple choice
Krajbich, I., Armel, C., & Rangel, A. (2010) · Nature Neuroscience · doi:10.1038/nn.2635 · Cited by 1,408
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304
Generating random correlation matrices based on vines and extended onion method
Lewandowski, D., Kurowicka, D., & Joe, H. (2009) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2009.04.008 · Cited by 1,135
Toward a rational and mechanistic account of mental effort
Shenhav, A., Musslick, S., Lieder, F., Kool, W., Griffiths, T. L., Cohen, J. D., & Botvinick, M. M. (2017) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-072116-031526 · Cited by 1,077
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 733
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693
Disorders of compulsivity: A common bias towards learning habits
Voon, V, Derbyshire, K., Rück, C., Irvine, M. A., Worbe, Y., Enander, J., Schreiber, L. R. N., Gillan, C., Fineberg, N. A., Sahakian, B. J., et al. (2015) · Molecular Psychiatry · doi:10.1038/mp.2014.44 · Cited by 643
Characterizing a psychiatric symptom dimension related to deficits in goal-directed control
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016) · eLife · doi:10.7554/elife.11305 · Cited by 586
Working-memory capacity protects model-based learning from stress
Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., & Daw, N. D. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1312011110 · Cited by 490
Redish, A. D. (2016) · Nature reviews. Neuroscience · doi:10.1038/nrn.2015.30 · Cited by 485
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 408
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357
Otto (2013) · doi:10.1177/0956797612463080 · Cited by 352
Dopamine transients are sufficient and necessary for acquisition of model-based associations
Sharpe, M. J., Chang, C. Y., Liu, M. A., Batchelor, H. M., Mueller, L. E., Jones, J. L., Niv, Y., & Schoenbaum, G. M. (2017) · Nature Neuroscience · doi:10.1038/nn.4538 · Cited by 298
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297
Parsimony, exhaustivity and balanced detection in neocortex
Romagnoni, A., Ribot, J., Bennequin, D., et al. (2016) · INRIA a CCSD electronic archive server · doi:10.1371/journal.pcbi · Cited by 297
Dopamine enhances model-based over model-free choice behavior
Wunderlich, K., Smittenaar, P., & Dolan, R. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.042 · Cited by 294
Model-based choices involve prospective neural activity
Doll, B. B., Duncan, K. D., Simon, D. A., Shohamy, D., & Daw, N. D. (2015) · Nature Neuroscience · doi:10.1038/nn.3981 · Cited by 289
Model-based learning protects against forming habits
Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0347-6 · Cited by 283
From Creatures of Habit to Goal-Directed Learners
Decker, J. H., Otto, A. R., Daw, N. D., & Hartley, C. A. (2016) · Psychological Science · doi:10.1177/0956797616639301 · Cited by 281
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280
Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems
Kool, W., Gershman, S. J., & Cushman, F. A. (2017) · Psychological Science · doi:10.1177/0956797617708288 · Cited by 255
Dorsal hippocampus contributes to model-based planning
Miller, K. J., Botvinick, M., & Brody, C. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4613 · Cited by 252
Smittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.009 · Cited by 244
Dezfouli, A., & Balleine, B. W. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003364 · Cited by 239
Deserno, L., Huys, Q. J. M., Boehme, R., Buchert, R., Heinze, H.-J., Grace, A. A., Dolan, R. J., Heinz, A., & Schlagenhauf, F. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1417219112 · Cited by 233
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217
When Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211
Cognitive Control Predicts Use of Model-based Reinforcement Learning
Otto, A. R., Skatova, A., Madlon‐Kay, S., & Daw, N. D. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00709 · Cited by 202
Model-Based and Model-Free Decisions in Alcohol Dependence
Sebold, M., Deserno, L., Nebe, S., Schad, D. J., Garbusow, M., Hägele, C., Keller, J., Jünger, E., Kathmann, N., Smolka, M. N., et al. (2014) · Neuropsychobiology · doi:10.1159/000362840 · Cited by 193
A Pallidus-Habenula-Dopamine Pathway Signals Inferred Stimulus Values
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., & Hikosaka, O. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00158.2010 · Cited by 186
Habitual control of goal selection in humans
Cushman, F. A., & Morris, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1506367112 · Cited by 172
Akam, T., Costa, R. M., & Dayan, P. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004648 · Cited by 155
Shahar, N., Hauser, T. U., Moutoussis, M., Moran, R., Keramati, M., & Dolan, R. J. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006803 · Cited by 152
Of goals and habits: age-related and individual differences in goal-directed decision-making
Eppinger, B., Walter, M., Heekeren, H. R., & Li, S. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00253 · Cited by 141
Habits as action sequences: hierarchical action control and changes in outcome value
Dezfouli, A., Lingawi, N. W., & Balleine, B. W. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0482 · Cited by 139
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 136
Holistic Reinforcement Learning: The Role of Structure and Attention
Radulescu, A., Niv, Y., & Ballard, I. C. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.01.010 · Cited by 109
Multiple memory systems as substrates for multiple decision systems
Doll, B. B., Shohamy, D., & Daw, N. D. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.04.014 · Cited by 101
Konovalov, A., & Krajbich, I. M. (2016) · Nature Communications · doi:10.1038/ncomms12438 · Cited by 98
Schad, D. J., Obst, E., Sebold, M., Garbusow, M., Bernhardt, N. R., Javadi, A., Zimmermann, U. S., Smolka, M. N., Heinz, A., Rapp, M. A., et al. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01450 · Cited by 92
Axiomatic methods, dopamine and reward prediction error
Caplin, A., & Dean, M. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.07.007 · Cited by 88
Model-Based Reasoning in Humans Becomes Automatic with Training
Economides, M., Kurth-Nelson, Z. L., Lübbert, A., Guitart‐Masip, M., & Dolan, R. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004463 · Cited by 74
Addiction as vulnerabilities in the decision process
Redish, A. D., Jensen, S., & Johnson, A. C. (2008) · Behavioral and Brain Sciences · doi:10.1017/s0140525x08004986 · Cited by 69
The successor representation in human reinforcement learningPreprint
Momennejad, I., Russek, E. M., Cheong, J., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/083824 · Cited by 69
Shahar, N., Moran, R., Hauser, T. U., Kievit, R., McNamee, D., Moutoussis, M., Dolan, R. J., Bullmore, E. T., Dolan, R. J., Goodyer, I. M., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1821647116 · Cited by 68
Planning Complexity Registers as a Cost in Metacontrol
Kool, W., Gershman, S. J., & Cushman, F. A. (2018) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01263 · Cited by 61
Daw, N. D. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0258-2 · Cited by 57
Hierarchical control of goal-directed action in the cortical–basal ganglia network
Balleine, B. W., Dezfouli, A., Ito, M., & Doya, K. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.06.001 · Cited by 51
Habits and goals: a motivational perspective on action control
Ceceli, A. O., & Tricomi, E. M. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.12.005 · Cited by 34
Toyama, A., Katahira, K., & Ohira, H. (2019) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2019.03.007 · Cited by 31
Model-Free RL or Action Sequences?
Morris, A., & Cushman, F. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02892 · Cited by 28
da Silva, C. F., & Hare, T. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195328 · Cited by 25
Self-citationHumans are primarily model-based learners in the two-stage taskPreprint
da Silva, C. F., & Hare, T. A. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/682922 · Cited by 17
Self-citationCan model-free reinforcement learning operate over information stored in working-memory?Preprint
da Silva, C. F., Yao, Y.-W., & Hare, T. A. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/107698 · Cited by 4
Self-citationda Silva, C. F., Yao, Y., & Hare, T. A. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-148191 · Cited by 4
Self-citation(2022) · doi:10.1162/jocn
Research works printed but not linked: 2
- Miller, K. J., Shenhav, A. & Ludvig, E. A. Habits without values. Psychol. Rev. 126, 292–311 (2019). · DOI · Paper page
- Economides, M., Kurth-Nelson, Z., Lübbert, A., Guitart-Masip, M. & Dolan, R. J. Model-based reasoning in humans becomes automatic with training. PLoS Comput. Biol. 11, e1004463 (2015). · DOI
Cited by
Sorted by citations
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
Reinforcement-learning in fronto-striatal circuits
Averbeck, B. B., & O’Doherty, J. P. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01108-0 · Cited by 132
A foundation model to predict and capture human cognition
Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éltető, N., et al. (2025) · Nature · doi:10.1038/s41586-025-09215-4 · Cited by 119
The interpretation of computational model parameters depends on the context
Eckstein, M. K., Master, S. L., Xia, L., Dahl, R. E., Wilbrecht, L., & Collins, A. G. E. (2022) · eLife · doi:10.7554/elife.75474 · Cited by 124
Vandaele, Y., & Ahmed, S. H. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-00899-y · Cited by 107
Metadata
- Authors
- Feher da Silva, C., & Hare, T. A.
- Affiliations
- University of Zurich (CH)
- Corresponding author
- Hare, Todd A.
- Venue
- Nature Human Behaviour
- Published
- 2020-07-06
- Volume
- 4
- Issue
- 10
- Pages
- 1053-1066
- ISSN
- 2397-3374 (ISSN-L)
- OpenAlex
- W3040440677
- Cited by
- 153
- References
- 64
- Citations by year
- 2026: 13 · 2025: 30 · 2024: 25 · 2023: 26 · 2022: 32 · 2021: 20 · 2020: 6
- FWCI
- 11.30
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- Nature Human Behaviour (publishedVersion)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
- Repository@Nottingham (University of Nottingham) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- model-based learning, model-free learning, task instructions, misconceptions, reward learning, habit formation, compulsivity, learning models, decision-making
- MeSH terms
- Adult, Choice Behavior, Goals, Habits, Humans, Models, Psychological, Probability Learning, Reinforcement, Psychology, Reward, Task Performance and Analysis, Thinking, Models, Statistical, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
