Retrospective revaluation in sequential decision making: a tale of two systems
Abstract
Recent computational theories of decision making in humans and animals have portrayed 2 systems locked in a battle for control of behavior. One system--variously termed model-free or habitual--favors actions that have previously led to reward, whereas a second--called the model-based or goal-directed system--favors actions that causally lead to reward according to the agent's internal model of the environment. Some evidence suggests that control can be shifted between these systems using neural or behavioral manipulations, but other evidence suggests that the systems are more intertwined than a competitive account would imply. In 4 behavioral experiments, using a retrospective revaluation design and a cognitive load manipulation, we show that human decisions are more consistent with a cooperative architecture in which the model-free system controls behavior, whereas the model-based system trains the model-free system by replaying and simulating experience.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hierarchical Active Inference predicts that responses should fail to be contextualized and become more habitual when the system is taxed by concurrent cognitively demanding tasks.
“For example, it predicts that responses should fail to be contextualized – and thus become more habitual – when the system is ‘taxed’ by concurrent cognitively demanding tasks, a phenomenon that has been often interpreted in terms of reinforcement learning theories (Gershman et al., 2014, Otto et al., 2013).”
From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Gershman et al., 2014
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 7,268 · see citing works
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641 · see citing works
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Working Memory Capacity as Executive Attention
Engle, R. W. (2002) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00160 · Cited by 2,507 · see citing works
Is working memory capacity task dependent?
Turner, M. L., & Engle, R. W. (1989) · Journal of Memory and Language · doi:10.1016/0749-596x(89)90040-5 · Cited by 2,299 · see citing works
An automated version of the operation span task
Unsworth, N., Heitz, R. P., Schrock, J. C., & Engle, R. W. (2005) · Behavior Research Methods · doi:10.3758/bf03192720 · Cited by 1,861 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Self-citationBalleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775 · see citing works
Reverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727 · see citing works
Sutton, R. S. (1990) · Elsevier eBooks · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,348 · see citing works
Working memory capacity and its relation to general intelligence
Conway, A. R. A., Kane, M. J., & Engle, R. W. (2003) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2003.10.005 · Cited by 1,223 · see citing works
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217 · see citing works
Forward and reverse hippocampal place-cell sequences during ripples
Diba, K., & Buzsáki, G. (2007) · Nature Neuroscience · doi:10.1038/nn1961 · Cited by 1,205 · see citing works
Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · see citing works
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · see citing works
Hippocampal replay is not a simple function of experience
Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S., & Redish, A. D. (2010) · Neuron · doi:10.1016/j.neuron.2010.01.034 · Cited by 733 · see citing works
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 724 · see citing works
Preplay of future place cell sequences by hippocampal cellular assemblies
Dragoi, G., & Tonegawa, S. (2011) · Nature · doi:10.1038/nature09633 · Cited by 716 · see citing works
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 635 · see citing works
Modulation of competing memory systems by distraction
Foerde, K., Knowlton, B. J., & Poldrack, R. A. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0602659103 · Cited by 600 · see citing works
Determining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529 · see citing works
Integrating Memories in the Human Brain: Hippocampal-Midbrain Encoding of Overlapping Events
Daphna Shohamy, A., & Wagner, A. D. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.023 · Cited by 525 · see citing works
Hippocampal mediation of stimulus representation: A computational theory
Gluck, M. A., & Myers, C. E. (1993) · Hippocampus · doi:10.1002/hipo.450030410 · Cited by 496 · see citing works
Cue Competition in Causality Judgments: The Role of Nonpresentation of Compound Stimulus Elements
Hamme, L. J. V., & Wasserman, E. A. (1994) · Learning and Motivation · doi:10.1006/lmot.1994.1008 · Cited by 491 · see citing works
The Role of the Hippocampus in Prediction and Imagination
Buckner, R. L. (2010) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163508 · Cited by 419 · see citing works
Within Compound Associations Mediate the Retrospective Revaluation of Causality Judgements
Dickinson, A., & Burke, J. (1996) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/713932614 · Cited by 417 · see citing works
Hippocampus leads ventral striatum in replay of place-reward information
Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. (2009) · PLoS Biology · doi:10.1371/journal.pbio.1000173 · Cited by 400 · see citing works
The Role of Learning in the Operation of Motivational Systems
Dickinson, A., & Balleine, B. W. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0312 · Cited by 387 · see citing works
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · see citing works
Self-citationForward and Backward Blocking in Human Contingency Judgement
Shanks, D. R. (1985) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748508402082 · Cited by 361 · see citing works
Otto (2013) · doi:10.1177/0956797612463080 · Cited by 352 · see citing works
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Information theory, novelty and hippocampal responses: unpredicted or unpredictable?
Strange, B. A., Duggins, A., Penny, W. D., Dolan, R. J., & Friston, K. J. (2005) · Neural Networks · doi:10.1016/j.neunet.2004.12.004 · Cited by 264 · see citing works
Integrating hippocampus and striatum in decision-making
Johnson, A. C., van der Meer, M. A. A., & Redish, A. D. (2007) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.01.003 · Cited by 254 · see citing works
Dual-task interference in perceptual category learning
Zeithamová, D., & Maddox, W. T. (2006) · Memory & Cognition · doi:10.3758/bf03193416 · Cited by 230 · see citing works
Trial order affects cue interaction in contingency judgment.
Chapman, G. B. (1991) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.17.5.837 · Cited by 185 · see citing works
Encoding uncertainty in the hippocampus
Harrison, L. M., Duggins, A., & Friston, K. J. (2006) · Neural Networks · doi:10.1016/j.neunet.2005.11.002 · Cited by 158 · see citing works
A triple dissociation of memory systems: Hippocampus, amygdala, and dorsal striatum.
McDonald, R., & White, N. (1993) · Behavioral Neuroscience · doi:10.1037/a0034883 · Cited by 146 · see citing works
Dissociating hippocampal and striatal contributions to sequential prediction learning
Bornstein, A. M., & Daw, N. D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07920.x · Cited by 120 · see citing works
Retrospective revaluation of causal judgments under positive and negative contingencies.
Larkin, M. J. W., Aitken, M. R. F., & Dickinson, A. (1998) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.24.6.1331 · Cited by 114 · see citing works
Johnson, A. C., & Redish, A. D. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.08.009 · Cited by 91 · see citing works
Reward modulates neuronal activity in the hippocampus of the rat
Hölscher, C., Jacob, W., & Mallot, H. A. (2003) · Behavioural Brain Research · doi:10.1016/s0166-4328(02)00422-9 · Cited by 90 · see citing works
Melchers, K. G., Lachnit, H., & Shanks, D. R. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000042 · Cited by 78 · see citing works
There are at least two kinds of probability matching: Evidence from a secondary task
Otto, A. R., Taylor, E. G., & Markman, A. B. (2010) · Cognition · doi:10.1016/j.cognition.2010.11.009 · Cited by 76 · see citing works
Self-citationLMS rules and the inverse base-rate effect: Comment on Gluck and Bower (1988).
Markman, A. B. (1989) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.118.4.417 · Cited by 61 · see citing works
Self-citationContext Learning in the Rodent Hippocampus
Fuhs, M. C., & Touretzky, D. S. (2007) · Neural Computation · doi:10.1162/neco.2007.19.12.3173 · Cited by 57 · see citing works
Interpreting hippocampal function as recoding and forecasting
Levy, W. B., Hocking, A. B., & Wu, X. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.08.005 · Cited by 49 · see citing works
Environmental statistics and the trade-off between model-based and TD learning in humans
Simon, D. A., & Daw, N. D. (2011) · neural information processing systems · Cited by 23 · see citing works
Dual learning processes in interactive skill acquisition.
Fu, W., & Anderson, J. R. (2008) · Journal of Experimental Psychology Applied · doi:10.1037/1076-898x.14.2.179 · Cited by 21 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550 · see citing works
Self-citationWood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445 · see citing works
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977 · see citing works
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · see citing works
“For example, it predicts that responses should fail to be contextualized – and thus become more habitual – when the system is ‘taxed’ by concurrent cognitively demanding tasks, a phenomenon that has been often interpreted in terms of reinforcement learning theories (Gershman et al., 2014; Otto et al., 2013).”
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Self-citationModel-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation
Dayan, P., & Berridge, K. C. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0277-8 · Cited by 357 · see citing works
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280 · see citing works
Deconstructing the human algorithms for exploration
Gershman, S. J. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.014 · Cited by 267 · see citing works
Self-citationSmittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.009 · Cited by 244 · see citing works
When Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211 · see citing works
Self-citationDepression: a decision-theoretic analysis
Huys, Q. J. M., Daw, N. D., & Dayan, P. (2015) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071714-033928 · Cited by 221 · see citing works
Internally generated sequences in learning and executing goal-directed behavior
Pezzulo, G., van der Meer, M. A. A., Lansink, C. S., & Pennartz, C. M. A. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.06.011 · Cited by 217 · see citing works
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · see citing works
Self-citationCognitive Control: Core Constructs and Current Considerations
Cohen, J. D. (2017) · The Wiley Handbook of Cognitive Control · doi:10.1002/9781118920497.ch1 · Cited by 179 · see citing works
Pezzulo, G., Rigoli, F., & Chersi, F. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00092 · Cited by 163 · see citing works
Humans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153 · see citing works
A psychoengineering paradigm for the neurocognitive mechanisms of biofeedback and neurofeedback
Gaume, A., Vialatte, A., Mora-Sánchez, A., Ramdani, C., & Vialatte, F. B. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.012 · Cited by 133 · see citing works
Incentives Boost Model-Based Control Across a Range of Severity on Several Psychiatric Constructs
Patzelt, E. H., Kool, W., Millner, A. J., & Gershman, S. J. (2019) · Biological Psychiatry · doi:10.1016/j.biopsych.2018.06.018 · Cited by 94 · see citing works
Self-citationA theory of learning to infer.
Dasgupta, I., Schulz, E., Tenenbaum, J. B., & Gershman, S. J. (2020) · Psychological Review · doi:10.1037/rev0000178 · Cited by 70 · see citing works
Self-citation
- Authors
- Gershman, S. J., Markman, A. B., & Otto, A. R.
- Affiliations
- Princeton University (US)
- The University of Texas at Austin (US)
- Venue
- Journal of Experimental Psychology General
- Published
- 2014
- Volume
- 143
- Issue
- 1
- Pages
- 182-194
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W2149892666
- Cited by
- 217
- References
- 52
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 25 · 2023: 11 · 2022: 17 · 2021: 22 · 2020: 20 · 2019: 14 · 2018: 18 · 2017: 22 · 2016: 15 · 2015: 17 · 2014: 9 · 2013: 11
- FWCI
- 6.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Mike Hogg Endowment
- National Science Foundation
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- model-free learning, model-based learning, goal-directed behavior, habitual behavior, sequential decision making, retrospective revaluation, cognitive load, internal models, reinforcement learning, human decision-making
- MeSH terms
- Decision Making, Humans, Models, Psychological, Neuropsychological Tests, Reinforcement, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
