Cortical and Hippocampal Correlates of Deliberation during Model-Based Decisions for Rewards in Humans
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
How do we use our memories of the past to guide decisions we've never had to make before? Although extensive work describes how the brain learns to repeat rewarded actions, decisions can also be influenced by associations between stimuli or events not directly involving reward - such as when planning routes using a cognitive map or chess moves using predicted countermoves - and these sorts of associations are critical when deciding among novel options. This process is known as model-based decision making. While the learning of environmental relations that might support model-based decisions is well studied, and separately this sort of information has been inferred to impact decisions, there is little evidence concerning the full cycle by which such associations are acquired and drive choices. Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally, or instead rely on other, specialized representations. Here, building on our previous work, which isolated dual representations underlying sequential predictive learning, we directly demonstrate that one such representation, encoded by the hippocampal memory system and adjacent cortical structures, supports goal-directed decisions. Using interleaved learning and decision tasks, we monitor predictive learning directly and also trace its influence on decisions for reward. We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits. Across both tasks, a quantitatively consistent learning process explains reaction times, choices, and both expectation- and surprise-related neural activity. The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions. These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Ratcliff, R. (1978) · Psychological Review · doi:10.1037/0033-295x.85.2.59 · Cited by 4,513 · cited 2×
“Motivated by race and sampling models [18], the model instantiates the decision variable on a particular trial by conducting some number n of”
MethodCircular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Buckner, R. L., & Carroll, D. C. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.11.004 · Cited by 2,691
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationThe role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
A framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
A cortical area selective for visual processing of the human body
Downing, P. E., Jiang, Y., Shuman, M., & Kanwisher, N. (2001) · Science · doi:10.1126/science.1063414 · Cited by 2,043
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Addis, D. R., Wong, A. T., & Schacter, D. L. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.10.016 · Cited by 1,949 · cited 1×
“It also joins those ideas with a literature suggesting that episodic memories can influence decisions [46,56,88].”
BackgroundThe cognitive neuroscience of constructive memory: remembering the past and imagining the future
Schacter, D. L., & Addis, D. R. (2007) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2007.2087 · Cited by 1,836 · cited 1×
“These proposals — along with observations of hippocampus and default network activity during look-ahead planning [75–77] — concord with our interpretation of the choice difficulty correlate as reflecting reinstatement of prior experiences.”
ResultA neostriatal habit learning system in humans
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996) · Science · doi:10.1126/science.273.5280.1399 · Cited by 1,777
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · cited 3×
“This same hippocampally-linked, slow process learning also matched the neural reward prediction error (RPE) in nucleus accumbens [29–31].”
ResultYin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226 · cited 1×
“The type of model-based decision making studied here contrasts with ‘‘model-free’’ habit learning, of the sort associated with dorsolateral striatum [58], predominant temporal-difference learning accounts of reward prediction error signal seen in dopamine neurons [6], and the striatal BOLD response [29–31].”
ResultModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Self-citationTracking the Hemodynamic Responses to Reward and Punishment in the Striatum
Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000) · Journal of Neurophysiology · doi:10.1152/jn.2000.84.6.3072 · Cited by 1,116 · cited 2×
“This same hippocampally-linked, slow process learning also matched the neural reward prediction error (RPE) in nucleus accumbens [29–31].”
ResultNeural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · cited 1×
“The idea we have advanced above, that successor states are retrieved stochastically (see also [45]), and their values integrated, connects directly with known neural mechanisms.”
BackgroundTemporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · cited 3×
“This same hippocampally-linked, slow process learning also matched the neural reward prediction error (RPE) in nucleus accumbens [29–31].”
ResultA triple dissociation of memory systems: Hippocampus, amygdala, and dorsal striatum.
McDonald, R. J., & White, N. M. (1993) · Behavioral Neuroscience · doi:10.1037/0735-7044.107.1.3 · Cited by 885
Competition among multiple memory systems: converging evidence from animal and human brain studies
Poldrack, R. A., & Packard, M. G. (2003) · Neuropsychologia · doi:10.1016/s0028-3932(02)00157-4 · Cited by 880 · cited 1×
“Further, the exclusive use of the slow-process associations in forward-looking, model-based choice suggest that these associations are of a type that may be flexibly recombined, a property long associated with hippocampal representations and not those of striatum [48,52,67].”
BackgroundPeters, J., & Büchel, C. (2010) · Neuron · doi:10.1016/j.neuron.2010.03.026 · Cited by 871 · cited 1×
“It also joins those ideas with a literature suggesting that episodic memories can influence decisions [46,56,88].”
BackgroundDynamic reorganization of striatal circuits during the acquisition and consolidation of a skill
Yin, H. H., Mulcare, S. P., Hilário, M. R. F., Clouse, E., Holloway, T., Davis, M. I., Hansson, A. C., Lovinger, D. M., & Costa, R. M. (2009) · Nature Neuroscience · doi:10.1038/nn.2261 · Cited by 818
Stewart, N., Chater, N., & Brown, G. D. A. (2006) · Cognitive Psychology · doi:10.1016/j.cogpsych.2005.10.003 · Cited by 737
Neural repetition suppression reflects fulfilled perceptual expectations
Summerfield, C., Trittschuh, E. H., Monti, J. M., Mesulam, M.-M., & Egner, T. (2008) · Nature Neuroscience · doi:10.1038/nn.2163 · Cited by 692 · cited 1×
“This idea is consistent with suggestions that anticipatory activity in category regions is driven by evidence accumulation [72].”
ResultThe representation of stimulus familiarity in anterior inferior temporal cortex
Li, L., Miller, E. K., & Desimone, R. (1993) · Journal of Neurophysiology · doi:10.1152/jn.1993.69.6.1918 · Cited by 539
Turk-Browne, N. B., Scholl, B. J., Chun, M. M., & Johnson, M. K. (2009) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21131 · Cited by 523
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Shaping of Object Representations in the Human Medial Temporal Lobe Based on Temporal Regularities
Schapiro, A. C., Kustner, L. V., & Turk-Browne, N. B. (2012) · Current Biology · doi:10.1016/j.cub.2012.06.056 · Cited by 430 · cited 1×
“among abstract stimuli [23].”
The Role of the Hippocampus in Prediction and Imagination
Buckner, R. L. (2010) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163508 · Cited by 419 · cited 1×
“Further, the exclusive use of the slow-process associations in forward-looking, model-based choice suggest that these associations are of a type that may be flexibly recombined, a property long associated with hippocampal representations and not those of striatum [48,52,67].”
BackgroundSchönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2496-07.2007 · Cited by 390
Self-citationSpeed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · cited 1×
“In particular, although the idea of model-based planning as a mnemonic version of evidence accumulation differs at least superficially from more abstract conceptualizations based on tree search [3,80,81] or Bayesian inference [82,83], sampling from successor states provides a more realizable process-level account of m…”
BackgroundMapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277 · cited 1×
“This method appears to perform robustly in this and our previous study [7] and other closely related analyses of parametric brain-behavior relationships [38,40,41], but there has not yet been a formal simulation study quantifying the error introduced by this approximation.”
MethodSelf-citationTracking the emergence of conceptual knowledge during human decision making
Kumaran, D., Summerfield, J. J., Hassabis, D., & Maguire, E. A. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.030 · Cited by 260 · cited 2×
“A suggestive connection of these ideas to nonspatial tasks is ubiquitous findings that the the hippocampal system is implicated in acquired equivalence, transitive inference, and sensory preconditioning effects [41,51–53], as well as the flexible use of conceptual [54] and structured [55] knowledge.”
Anterior hippocampus and goal-directed spatial decision making
Viard, A., Doeller, C. F., Hartley, T., Bird, C. M., & Burgess, N. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4640-10.2011 · Cited by 144
Gershman, S. J., Pesaran, B., & Daw, N. D. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2469-09.2009 · Cited by 118 · cited 1×
“For the former, in our experience [22,25,38,92–95], enforcing common model parameters provides a simple regularization that improves the reliability of populationlevel neural results.”
BackgroundSelf-citationGeneralization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · cited 2×
“This method appears to perform robustly in this and our previous study [7] and other closely related analyses of parametric brain-behavior relationships [38,40,41], but there has not yet been a formal simulation study quantifying the error introduced by this approximation.”
BackgroundMethodSelf-citation
Cited by
Decision Making and Sequential Sampling from Memory
Shadlen, M. N., & Shohamy, D. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.036 · Cited by 418
Hippocampal theta sequences reflect current goals
Wikenheiser, A. M., & Redish, A. D. (2015) · Nature Neuroscience · doi:10.1038/nn.3909 · Cited by 367
A map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex
Wikenheiser, A. M., & Schoenbaum, G. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.56 · Cited by 359
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280
“To the extent that model-based forms of decision-making draw on additional mechanisms, including hippocampally-mediated stimulus-stimulus associations (Bornstein & Daw, 2013; Bunsey & Eichenbaum, 1996; Dusek & Eichenbaum, 1997) and prefrontal control mechanisms (E.”
Background
Metadata
- Authors
- Bornstein, A. M., & Daw, N. D.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-12-05
- Volume
- 9
- Issue
- 12
- Pages
- e1003387
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2146289716
- Cited by
- 91
- References
- 39
- FWCI
- 3.03
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.