Policy Adjustment in a Dynamic Economic Game
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Making sequential decisions to harvest rewards is a notoriously difficult problem. One difficulty is that the real world is not stationary and the reward expected from a contemplated action may depend in complex ways on the history of an animal's choices. Previous functional neuroimaging work combined with principled models has detected brain responses that correlate with computations thought to guide simple learning and action choice. Those works generally employed instrumental conditioning tasks with fixed action-reward contingencies. For real-world learning problems, the history of reward-harvesting choices can change the likelihood of rewards collected by the same choices in the near-term future. We used functional MRI to probe brain and behavioral responses in a continuous decision-making task where reward contingency is a function of both a subject's immediate choice and his choice history. In these more complex tasks, we demonstrated that a simple actor-critic model can account for both the subjects' behavioral and brain responses, and identified a reward prediction error signal in ventral striatal structures active during these non-stationary decision tasks. However, a sudden introduction of new reward structures engages more complex control circuitry in the prefrontal cortex (inferior frontal gyrus and anterior insula) and is not captured by a simple actor-critic model. Taken together, these results extend our knowledge of reward-learning signals into more complex, history-dependent choice tasks. They also highlight the important interplay between striatum and prefrontal cortex as decision-makers respond to the strategic demands imposed by non-stationary reward environments more reminiscent of real-world tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An fMRI Investigation of Emotional Engagement in Moral Judgment
Greene, J. D., Sommerville, R. B., Nystrom, L. E., Darley, J. M., & Cohen, J. D. (2001) · Science · doi:10.1126/science.1062872 · Cited by 4,561
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089
Self-citationDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Both of Us Disgusted in My Insula
Wicker, B., Keysers, C., Plailly, J., Royet, J.-P., Gallese, V., & Rizzolatti, G. (2003) · Neuron · doi:10.1016/s0896-6273(03)00679-2 · Cited by 2,185
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“In this paper, we study changes in action selection reflective of changes in reward expectation in a series of tasks in which earned reward depends in complex ways on previous actions. The rationale of this study was to fit each subject's behavior through time with a continuous error-based learning model (e.g. actor-c…”
MethodSelf-citationThe prefontral cortex and cognitive control
Miller, E. K. (2000) · Nature Reviews Neuroscience · doi:10.1038/35036228 · Cited by 2,016 · cited 3×
“Numerous recent reports have identified the striatum and prefrontal cortex as two parallel, and often times competing, systems that interact to guide behavior [25–26,33,35,39–42].”
BackgroundBy Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Cited by
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 717
Anterior insular cortex and emotional awareness
Gu, X., Hof, P. R., Friston, K. J., & Fan, J. (2013) · Journal of Comparative Neurology · doi:10.1002/cne.23368 · Cited by 712
“Diminished ability to integrate information rapidly among spatially distinct regions may underlie functional deficits in these conditions and, ultimately, in the inability to make quick and intuitive judgments regarding uncertain and rapidly changing social contexts (Allman et al., 2005).”
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 704
Trial-by-trial data analysis using computational models
Daw, N. D. (2011) · Decision Making, Affect, and Learning · doi:10.1093/acprof:oso/9780199600434.003.0001 · Cited by 456
Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 438
Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain
Niv, Y., Edlund, J. A., Dayan, P., & O'Doherty, J. P. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5498-10.2012 · Cited by 356
“Numerous previous fMRI studies employing whole-brain analysis methods have shown that BOLD activity in the ventral striatum of humans engaged in classical or instrumental conditioning is correlated with prediction errors (McClure et al., 2003; O’Doherty et al., 2003, 2004; Abler et al., 2006; Li et al., 2006; Preuscho…”
BackgroundInfluentialDopaminergic-neuropeptide interactions in the social brain
Skuse, D. H., & Gallagher, L. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.09.007 · Cited by 340
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277
“We fit participants’ choices using a temporal-difference learning model (Sutton and Barto, 1998) similar to those used to account for choice behavior and neural signals in previous studies (Daw et al., 2006; Li et al., 2006; Tanaka et al., 2004).”
MethodTesting the Reward Prediction Error Hypothesis with an Axiomatic Model
Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1747-10.2010 · Cited by 244
“…magnetic resonance imaging (fMRI) measurements of neural activity in dopamine target areas (McClure et al., 2003; O’Doherty et al., 2003, 2004; Seymour et al., 2004; Abler et al., 2006; Li et al., 2006; Pessiglione et al., 2006; Behrens et al., 2008; D’Ardenne et al., 2008; Hare et al., 2008).”
BackgroundNeural signature of fictive learning signals in a sequential investment task
Lohrenz, T., McCabe, K., Camerer, C. F., & Montague, P. R. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608842104 · Cited by 265
Experimental subjects are not different
Exadaktylos, F., Espín, A. M., & Brañas-Garza, P. (2013) · Scientific Reports · doi:10.1038/srep01213 · Cited by 257
Reward-related processing in the human brain: developmental considerations
Fareri, D. S., Martin, L. N., & Delgado, M. R. (2008) · Development and Psychopathology · doi:10.1017/s0954579408000576 · Cited by 141
“Subsequent PET studies have also observed similar results during tasks investigating responses to monetary rewards in other types of games, food rewards (Volkow et al., 2002) and even symbolic feedback (Thut et al., 1997).”
ResultNeural correlates of risk prediction error during reinforcement learning in humans
d'Acremont, M., Lu, Z.-L., Li, X., Van der Linden, M., & Bechara, A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.04.096 · Cited by 94
Axiomatic methods, dopamine and reward prediction error
Caplin, A., & Dean, M. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.07.007 · Cited by 88
Neural Correlates of Reinforcement Learning and Social Preferences in Competitive Bidding
van den Bos, W., Talwar, A., & McClure, S. M. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.3095-12.2013 · Cited by 80
“Following others (Montague et al., 1996; O’Doherty et al., 2003; Li et al., 2006), we identified brain regions related to valuation by identifying areas that correlated with the prediction errors that underlie learning (Eqs.”
BackgroundSelf-citationDissociable reward and timing signals in human midbrain and ventral striatum
Klein-Flügge, M. C., Hunt, L. T., Bach, D. R., Dolan, R. J., & Behrens, T. E. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.08.024 · Cited by 75
“We note that the findings we present here are not inconsistent with the existence of a VS reward prediction error signal, even a dopaminergic one, in the many situations where subjects’ aim is indeed to maximize the occurrence and magnitude of accumulated rewards (Yacubian et al., 2006; Pessiglione et al., 2006; Harun…”
BackgroundResultStewart, J. L., Connolly, C. G., May, A. C., Tapert, S. F., Wittmann, M., & Paulus, M. P. (2014) · Addiction · doi:10.1111/add.12403 · Cited by 68
“This process involves the striatum, which implements contingencies linking choices with rewarding outcomes (46–48), and the insula and inferior frontal gyrus (IFG), regions that encode change in reward and risk variance during reinforcement (49, 50).”
Background
Metadata
- Authors
- Li, J., McClure, S. M., King-Casas, B., & Read Montague, P.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2006-12-20
- Volume
- 1
- Issue
- 1
- Pages
- e103
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2149774750
- Cited by
- 63
- References
- 28
- FWCI
- 3.69
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.