Policy Adjustment in a Dynamic Economic Game
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Making sequential decisions to harvest rewards is a notoriously difficult problem. One difficulty is that the real world is not stationary and the reward expected from a contemplated action may depend in complex ways on the history of an animal's choices. Previous functional neuroimaging work combined with principled models has detected brain responses that correlate with computations thought to guide simple learning and action choice. Those works generally employed instrumental conditioning tasks with fixed action-reward contingencies. For real-world learning problems, the history of reward-harvesting choices can change the likelihood of rewards collected by the same choices in the near-term future. We used functional MRI to probe brain and behavioral responses in a continuous decision-making task where reward contingency is a function of both a subject's immediate choice and his choice history. In these more complex tasks, we demonstrated that a simple actor-critic model can account for both the subjects' behavioral and brain responses, and identified a reward prediction error signal in ventral striatal structures active during these non-stationary decision tasks. However, a sudden introduction of new reward structures engages more complex control circuitry in the prefrontal cortex (inferior frontal gyrus and anterior insula) and is not captured by a simple actor-critic model. Taken together, these results extend our knowledge of reward-learning signals into more complex, history-dependent choice tasks. They also highlight the important interplay between striatum and prefrontal cortex as decision-makers respond to the strategic demands imposed by non-stationary reward environments more reminiscent of real-world tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089
Self-citationDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Both of Us Disgusted in My Insula
Wicker, B., Keysers, C., Plailly, J., Royet, J.-P., Gallese, V., & Rizzolatti, G. (2003) · Neuron · doi:10.1016/s0896-6273(03)00679-2 · Cited by 2,185
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“In this paper, we study changes in action selection reflective of changes in reward expectation in a series of tasks in which earned reward depends in complex ways on previous actions. The rationale of this study was to fit each subject's behavior through time with a continuous error-based learning model (e.g. actor-c…”
MethodSelf-citationThe prefontral cortex and cognitive control
Miller, E. K. (2000) · Nature Reviews Neuroscience · doi:10.1038/35036228 · Cited by 2,016 · cited 3×
“Numerous recent reports have identified the striatum and prefrontal cortex as two parallel, and often times competing, systems that interact to guide behavior [25–26,33,35,39–42].”
BackgroundBy Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans
Aron, A. R., Fletcher, P. C., Bullmore, E. T., Sahakian, B. J., & Robbins, T. W. (2003) · Nature Neuroscience · doi:10.1038/nn1003 · Cited by 1,771 · cited 1×
“The IFG, by contrast, is much more strongly linked to maintaining and switching between action strategies [ 37 ].”
BackgroundGoal-directed instrumental action: contingency and incentive learning and their cortical substrates
Balleine, B. W., & Dickinson, A. (1998) · Neuropharmacology · doi:10.1016/s0028-3908(98)00033-1 · Cited by 1,541
Cited by
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792
Anterior insular cortex and emotional awareness
Gu, X., Hof, P. R., Friston, K. J., & Fan, J. (2013) · Journal of Comparative Neurology · doi:10.1002/cne.23368 · Cited by 712
“Diminished ability to integrate information rapidly among spatially distinct regions may underlie functional deficits in these conditions and, ultimately, in the inability to make quick and intuitive judgments regarding uncertain and rapidly changing social contexts (Allman et al., 2005).”
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 717
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 704
Trial-by-trial data analysis using computational models
Daw, N. D. (2011) · Decision Making, Affect, and Learning · doi:10.1093/acprof:oso/9780199600434.003.0001 · Cited by 456
Metadata
- Authors
- Li, J., McClure, S. M., King-Casas, B., & Read Montague, P.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2006-12-20
- Volume
- 1
- Issue
- 1
- Pages
- e103
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2149774750
- Cited by
- 63
- References
- 25
- FWCI
- 3.69
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Policy Adjustment in a Dynamic Economic Game. https://app.cochat.ai/papers/10.1371/journal.pone.0000103
- The paper
- Li, J., McClure, S. M., King-Casas, B., & Read Montague, P. (2006). Policy Adjustment in a Dynamic Economic Game. PLoS ONE. https://doi.org/10.1371/journal.pone.0000103
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.