Multiple memory systems as substrates for multiple decision systems
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
It has recently become widely appreciated that value-based decision making is supported by multiple computational strategies. In particular, animal and human behavior in learning tasks appears to include habitual responses described by prominent model-free reinforcement learning (RL) theories, but also more deliberative or goal-directed actions that can be characterized by a different class of theories, model-based RL. The latter theories evaluate actions by using a representation of the contingencies of the task (as with a learned map of a spatial maze), called an “internal model.” Given the evidence of behavioral and neural dissociations between these approaches, they are often characterized as dissociable learning systems, though they likely interact and share common mechanisms. In many respects, this division parallels a longstanding dissociation in cognitive neuroscience between multiple memory systems, describing, at the broadest level, separate systems for declarative and procedural learning. Procedural learning has notable parallels with model-free RL: both involve learning of habits and both are known to depend on parts of the striatum. Declarative memory, by contrast, supports memory for single events or episodes and depends on the hippocampus. The hippocampus is thought to support declarative memory by encoding temporal and spatial relations among stimuli and thus is often referred to as a relational memory system. Such relational encoding is likely to play an important role in learning an internal model, the representation that is central to model-based RL. Thus, insofar as the memory systems represent more general-purpose cognitive mechanisms that might subserve performance on many sorts of tasks including decision making, these parallels raise the question whether the multiple decision systems are served by multiple memory systems, such that one dissociation is grounded in the other. Here we investigated the relationship between model-based RL and relational memory by comparing individual differences across behavioral tasks designed to measure either capacity. Human subjects performed two tasks, a learning and generalization task (acquired equivalence) which involves relational encoding and depends on the hippocampus; and a sequential RL task that could be solved by either a model-based or model-free strategy. We assessed the correlation between subjects’ use of flexible, relational memory, as measured by generalization in the acquired equivalence task, and their differential reliance on either RL strategy in the decision task. We observed a significant positive relationship between generalization and model-based, but not model-free, choice strategies. These results are consistent with the hypothesis that model-based RL, like acquired equivalence, relies on a more general-purpose relational memory system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
The Medial Temporal Lobe and Recognition Memory
Eichenbaum, H., Yonelinas, A. P., & Ranganath, C. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.30.051606.094328 · Cited by 2,580
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationCohen, N. J., & Squire, L. R. (1980) · Science · doi:10.1126/science.7414331 · Cited by 2,296 · cited 1×
“…in amnestics with varying etiologies in numerous incrementally acquired, feedback-driven procedural learning tasks, such as mirror-reading (Cohen & Squire, 1980; Gabrieli, Corkin, Mickel, & Growdon, 1993), the pursuit-rotor (Heindel, Salmon, Shults, Walicke, & Butters, 1989), and weather…”
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 2×
“In particular, model-free RL theories (Montague et al., 1996) maintain an estimate of the net reward previously received following each action, updating it in light of each experience using a reward prediction error signal.”
BackgroundMethodAddis, D. R., Wong, A. T., & Schacter, D. L. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.10.016 · Cited by 1,949
A neostriatal habit learning system in humans
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996) · Science · doi:10.1126/science.273.5280.1399 · Cited by 1,777
Patients with hippocampal amnesia cannot imagine new experiences
Hassabis, D., Kumaran, D., Vann, S. D., & Maguire, E. A. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0610561104 · Cited by 1,414
Phasic Firing in Dopaminergic Neurons Is Sufficient for Behavioral Conditioning
Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., & Deisseroth, K. (2009) · Science · doi:10.1126/science.1168878 · Cited by 1,266 · cited 1×
“Optogenetic work indicates a causal role for this signaling pathway in reward learning, consistent with the predictions of model-free RL (Steinberg et al., 2013; Tsai et al., 2009).”
ResultYin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226
Hippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Self-citationThe role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099
Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · cited 2×
“This parallel is further supported by suggestive, though mixed, evidence for hippocampal involvement in modelbased RL (Corbit & Balleine, 2000; Corbit, Ostlund, & Balleine, 2002; Johnson & Redish, 2007; Simon & Daw, 2011).”
ResultUnderstanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis
Glimcher, P. W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014269108 · Cited by 955 · cited 2×
“These theories explain the behavioral phenomena of habits (Daw, Niv, & Dayan, 2005)—inflexible response tendencies that often arise after overtraining (Adams, 1982)—and, neurally, offer the predominant computational account of the reward prediction error-like phasic responses of dopamine neurons and of similar signals…”
BackgroundPrediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · cited 2×
“…and ventromedial prefrontal areas), it has been implicated in several studies with elaborated task preparations (Lee, Ghim, Kim, Lee, & Jung, 2012; Tanaka et al., 2004; Wimmer, Daw, & Shohamy, 2012; Wimmer & Shohamy, 2012), including those formally examining model-based decisions (Bornstein &…”
BackgroundResultA specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · cited 2×
“Many experiments suggest that these two learning approaches trade off differentially, depending on task features such as the amount of training (Adams, 1982; Tricomi et al., 2009), dual-task interference (Otto et al.”
BackgroundHikida, T., Kimura, K., Wada, N., Funabiki, K., & Nakanishi, S. (2010) · Neuron · doi:10.1016/j.neuron.2010.05.011 · Cited by 592 · cited 1×
“Dopaminergic projections to striatum modulate activation and plasticity (Reynolds and Wickens, 2002) of corticostriatal synapses, driving reward and punishment learning (Hikida et al., 2010).”
BackgroundTurk-Browne, N. B., Scholl, B. J., Chun, M. M., & Johnson, M. K. (2009) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21131 · Cited by 523
Shaping of Object Representations in the Human Medial Temporal Lobe Based on Temporal Regularities
Schapiro, A. C., Kustner, L. V., & Turk-Browne, N. B. (2012) · Current Biology · doi:10.1016/j.cub.2012.06.056 · Cited by 430 · cited 1×
“Notably, this region does not merely represent space, but also appears to be involved in learning statistical relationships between stochastic events in general (Turk-Browne et al., 2009; Staresina and Davachi, 2009; Schapiro et al., 2012; Bornstein and Daw, 2012; Wimmer and Shohamy, 2012) and with drawing on memory f…”
BackgroundThe ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Self-citationOtto (2013) · doi:10.1177/0956797612463080 · Cited by 352
Dopamine enhances model-based over model-free choice behavior
Wunderlich, K., Smittenaar, P., & Dolan, R. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.042 · Cited by 294 · cited 1×
“2006), pharmacological interventions (Dickinson et al., 2000; Wunderlich et al., 2012) or brain lesions affecting areas associated with either system (Yin et al.”
BackgroundMind the Gap: Binding Experiences across Space and Time in the Human Hippocampus
Staresina, B. P., & Davachi, L. (2009) · Neuron · doi:10.1016/j.neuron.2009.06.024 · Cited by 273 · cited 1×
“…involved in learning statistical relationships between stochastic events in general (Bornstein & Daw, 2012; Schapiro, Kustner, & Turk-Browne, 2012; Staresina & Davachi, 2009; Turk-Browne, Scholl, Chun, & Johnson, 2009; Wimmer & Shohamy, 2012) and with drawing on memory for imagining future events…”
BackgroundThe hippocampus and inferential reasoning: building memories to navigate future decisions
Zeithamova, D., Schlichting, M. L., & Preston, A. R. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00070 · Cited by 266
Dissociating Hippocampal versus Basal Ganglia Contributions to Learning and Transfer
Myers, C. E., Shohamy, D., Gluck, M. A., Grossman, S., Kluger, A., Ferris, S., Golomb, J., Schnirman, G., & Schwartz, R. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321208123 · Cited by 243 · cited 7×
“, 2007), and specifically in supporting flexible generalization in acquired equivalence (Myers et al., 2003; Shohamy and Wagner, 2008) and similar tasks, such as associative inference and sensory preconditioning (Preston et al.”
BackgroundMethodInfluentialRetrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217 · cited 1×
“In contrast, a number of variant algorithms pre-compute this search; for example, DYNA and prioritized sweeping (Sutton, 1990; Moore and Atkeson, 1993), though these have received less attention in psychology and neuroscience (Gershman et al., 2012).”
BackgroundGeneralization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · cited 1×
“Although the hippocampus surfaces relatively rarely in the literature on RL and decision making (compared to striatal and ventromedial prefrontal areas), it has been implicated in several studies with elaborated task preparations (Tanaka et al., 2004; Lee et al., 2012; Wimmer et al., 2012; Wimmer and Shohamy, 2012), i…”
ResultSelf-citation
Cited by
Zilverstand, A., Huang, A. S., Alia-Klein, N., & Goldstein, R. Z. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.048 · Cited by 562
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Self-citationOver the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex
Wikenheiser, A. M., & Schoenbaum, G. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.56 · Cited by 359
Model-based choices involve prospective neural activity
Doll, B. B., Duncan, K. D., Simon, D. A., Shohamy, D., & Daw, N. D. (2015) · Nature Neuroscience · doi:10.1038/nn.3981 · Cited by 283
Self-citationHumans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153
Metadata
- Authors
- Doll, B. B., Shohamy, D., & Daw, N. D.
- Venue
- Neurobiology of Learning and Memory
- Publisher
- Elsevier BV
- Published
- 2015-01
- Volume
- 117
- Pages
- 4-13
- ISSN
- 1074-7427 (ISSN-L)
- OpenAlex
- W2108668231
- Cited by
- 95
- References
- 28
- FWCI
- 4.77
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.