Valuation of uncertain and delayed rewards in primate prefrontal cortex
Open accessAbstract
Humans and animals often must choose between rewards that differ in their qualities, magnitudes, immediacy, and likelihood, and must estimate these multiple reward parameters from their experience. However, the neural basis for such complex decision making is not well understood. To understand the role of the primate prefrontal cortex in determining the subjective value of delayed or uncertain reward, we examined the activity of individual prefrontal neurons during an inter-temporal choice task and a computer-simulated competitive game. Consistent with the findings from previous studies in humans and other animals, the monkey's behaviors during inter-temporal choice were well accounted for by a hyperbolic discount function. In addition, the activity of many neurons in the lateral prefrontal cortex reflected the signals related to the magnitude and delay of the reward expected from a particular action, and often encoded the difference in temporally discounted values that predicted the animal's choice. During a computerized matching pennies game, the animals approximated the optimal strategy, known as Nash equilibrium, using a reinforcement learning algorithm. We also found that many neurons in the lateral prefrontal cortex conveyed the signals related to the animal's previous choices and their outcomes, suggesting that this cortical area might play an important role in forming associations between actions and their outcomes. These results show that the primate lateral prefrontal cortex plays a central role in estimating the values of alternative actions based on multiple sources of information.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A few studies have estimated the time constant of the reward filter from behavior, and assessed how past rewards affect choice selection.
“So far a few studies have estimated the time constant of this filter from behavior, and assessed how past rewards affect choice selection8-12”
From A reservoir of time constants for memory traces in cortical neurons (2011) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,700 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Theory of games and economic behavior
Garcia, S. R., & Miller, S. J. (2019) · American Mathematical Society eBooks · doi:10.1090/mbk/121/32 · Cited by 16,777 · see citing works
Time Discounting and Time Preference: A Critical Review
Frederick, S., Loewenstein, G., & O’Donoghue, T. (2002) · Journal of Economic Literature · doi:10.1257/002205102320161311 · Cited by 6,106 · see citing works
The Theory of Games and Economic Behavior
Harrison, R. W., von Neumann, J., & Morgenstern, O. (1945) · Journal of Farm Economics · doi:10.2307/1232672 · Cited by 5,258 · see citing works
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564 · see citing works
Behavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,801 · see citing works
Fehr, E., & Fischbacher, U. (2003) · Nature · doi:10.1038/nature02043 · Cited by 3,475 · cited 1× · see citing works
“This might result because human decision makers are sometimes altruistic and therefore choose actions that can increase the well-beings of others at the expense of their own (Fehr & Camerer, 2007; Fehr & Fischbacher, 2003).”
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089 · see citing works
An adjusting procedure for studying delayed reinforcement.
Mazur, J. E. (1987) · doi:10.4324/9781315825502-11 · Cited by 2,189 · see citing works
The neural correlates of subjective value during intertemporal choice
Kable, J. W., & Glimcher, P. W. (2007) · Nature Neuroscience · doi:10.1038/nn2007 · Cited by 1,916 · see citing works
Developments in Non-Expected Utility Theory: The Hunt for a Descriptive Theory of Choice under Risk
Starmer, C. (2000) · Journal of Economic Literature · doi:10.1257/jel.38.2.332 · Cited by 1,852 · see citing works
Erev, I., & Roth, A. E. (1998) · American Economic Review · Cited by 1,794 · see citing works
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721 · see citing works
A Discounting Framework for Choice With Delayed and Probabilistic Rewards.
Green, L., & Myerson, J. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.5.769 · Cited by 1,676 · see citing works
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598 · see citing works
SUBJECTIVE PROBABILITY AND DELAY
Rachlin, H., Raineri, A., & Cross, D. (1991) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1991.55-233 · Cited by 1,341 · see citing works
Behavioral game theory, experiments in strategic interaction
Canté, F. (2004) · Cuadernos de Economía · Cited by 1,265 · see citing works
COMMITMENT, CHOICE AND SELF‐CONTROL<sup>1</sup>
Rachlin, H., & Green, L. (1972) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1972.17-15 · Cited by 1,204 · see citing works
Discounting of Delayed Rewards: A Life-Span Comparison
Green, L., Fry, A. F., & Myerson, J. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00610.x · Cited by 1,165 · see citing works
In all likelihood : statistical modelling and inference using likelihood
Pawitan, Y. (2001) · Cited by 1,068 · see citing works
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030 · see citing works
Time Discounting for Primary Rewards
McClure, S. M., Ericson, K. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.4246-06.2007 · Cited by 971 · see citing works
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · see citing works
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866 · see citing works
DISCOUNTING OF DELAYED REWARDS: MODELS OF INDIVIDUAL CHOICE
Myerson, J., & Green, L. (1995) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1995.64-263 · Cited by 843 · see citing works
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · see citing works
Social neuroeconomics: the neural circuitry of social preferences
Fehr, E., & Camerer, C. F. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.002 · Cited by 779 · see citing works
Bidding on the future: Evidence against normative discounting of delayed rewards.
Kirby, K. N. (1997) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.126.1.54 · Cited by 750 · see citing works
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 690 · see citing works
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678 · see citing works
Self-citationDelay discounting of real and hypothetical rewards.
Madden, G. J., Begotka, A. M., Raiff, B. R., & Kastern, L. L. (2003) · Experimental and Clinical Psychopharmacology · doi:10.1037/1064-1297.11.2.139 · Cited by 660 · see citing works
Expectation of reward modulates cognitive signals in the basal ganglia
Kawagoe, R., Takikawa, Y., & Hikosaka, O. (1998) · Nature Neuroscience · doi:10.1038/1625 · Cited by 632 · see citing works
Intertemporal choice – toward an integrative framework
Berns, G. S., Laibson, D., & Loewenstein, G. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.08.011 · Cited by 580 · see citing works
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498 · see citing works
The computational neurobiology of learning and reward
Daw, N. D., & Doya, K. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.006 · Cited by 462 · see citing works
Dissociable neural representations of future reward magnitude and delay during temporal discounting
Ballard, K., & Knutson, B. D. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.11.004 · Cited by 450 · see citing works
Leon, M. I., & Shadlen, M. N. (1999) · Neuron · doi:10.1016/s0896-6273(00)80854-5 · Cited by 449 · see citing works
Modeling Myopic Decisions: Evidence for Hyperbolic Delay-Discounting within Subjects and Amounts
Kirby, K. N., & Maraković, N. N. (1995) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1995.1086 · Cited by 448 · see citing works
Medial prefrontal cell activity signaling prediction errors of action values
Matsumoto, M., Matsumoto, K., Abe, H., & Tanaka, K. (2007) · Nature Neuroscience · doi:10.1038/nn1890 · Cited by 440 · see citing works
Dorris, M. C., & Glimcher, P. W. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.009 · Cited by 428 · see citing works
Value Representations in the Primate Striatum during Matching Behavior
Lau, B., & Glimcher, P. W. (2008) · Neuron · doi:10.1016/j.neuron.2008.02.021 · Cited by 411 · see citing works
Preference reversal and delayed reinforcement
Ainslie, G. W., & Herrnstein, R. J. (1981) · Animal Learning & Behavior · doi:10.3758/bf03209777 · Cited by 382 · see citing works
Influence of Reward Expectation on Behavior-Related Neuronal Activity in Primate Striatum
Hollerman, J. R., Tremblay, L., & Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.2.947 · Cited by 380 · see citing works
Rosati, A. G., Stevens, J. R., Hare, B. A., & HAUSER, M. D. (2007) · Current Biology · doi:10.1016/j.cub.2007.08.033 · Cited by 357 · see citing works
Preference reversal and self control: choice as a function of reward amount and delay
Green, L., Fisher, E. B., Perlow, S., & Sherman, L. (1981) · Cited by 335 · see citing works
Discounting of Monetary and Directly Consumable Rewards
Estle, S. J., Green, L., Myerson, J., & Holt, D. D. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01849.x · Cited by 328 · see citing works
Seo, H., & Lee, D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2369-07.2007 · Cited by 324 · see citing works
Self-citationGame theory and neural basis of social decision making
Lee, D. (2008) · Nature Neuroscience · doi:10.1038/nn2065 · Cited by 296 · see citing works
Self-citationThe ecology and evolution of patience in two New World monkeys
Stevens, J. R., Hallinan, E. V., & HAUSER, M. D. (2005) · Biology Letters · doi:10.1098/rsbl.2004.0285 · Cited by 289 · see citing works
Neural signature of fictive learning signals in a sequential investment task
Lohrenz, T. M., McCabe, K., Camerer, C. F., & Montague, P. R. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608842104 · Cited by 263 · see citing works
Learning Behavior in an Experimental Matching Pennies Game
Mookherjee, D., & Sopher, B. (1994) · Games and Economic Behavior · doi:10.1006/game.1994.1037 · Cited by 259 · see citing works
Prefrontal Coding of Temporally Discounted Values during Intertemporal Choice
Kim, S., Hwang, J., & Lee, D. (2008) · Neuron · doi:10.1016/j.neuron.2008.05.010 · Cited by 241 · see citing works
Self-citationIs a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making
Tobias Kalenscher, A., & Pennartz, C. M. A. (2007) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2007.11.004 · Cited by 222 · see citing works
Moment-to-moment tracking of state value in the amygdala
Belova, M. A., Paton, J. J., & Salzman, C. D. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1400-08.2008 · Cited by 221 · see citing works
Reliability of a Measure of Temporal Discounting
Simpson, C. A., & Vuchinich, R. E. (2000) · The Psychological Record · doi:10.1007/bf03395339 · Cited by 181 · see citing works
The neural substrates of probabilistic and intertemporal decision making
Weber, B. J., & Huettel, S. A. (2008) · Brain Research · doi:10.1016/j.brainres.2008.07.105 · Cited by 179 · see citing works
Action and Outcome Encoding in the Primate Caudate Nucleus
Lau, B., & Glimcher, P. W. (2007) · Journal of Neuroscience · doi:10.1523/jneurosci.3060-07.2007 · Cited by 177 · see citing works
Reinforcement learning and decision making in monkeys during a competitive game
Lee, D., Conroy, M. L., McGreevy, B. P., & Barraclough, D. J. (2004) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.07.007 · Cited by 156 · see citing works
Self-citationNeural Dissociation of Delay and Uncertainty in Intertemporal Choice
Luhmann, C. C., Chun, M. M., Yi, D.-J., Lee, D., & Wang, X. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.5058-08.2008 · Cited by 150 · see citing works
Self-citationDynamic Signals Related to Choices and Outcomes in the Dorsolateral Prefrontal Cortex
Seo, H., Barraclough, D. J., & Lee, D. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhm064 · Cited by 143 · see citing works
Self-citationBehavioral and Neural Changes after Gains and Losses of Conditioned Reinforcers
Seo, H., & Lee, D. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.4726-08.2009 · Cited by 137 · see citing works
Self-citationLearning and decision making in monkeys during a rock–paper–scissors game
Lee, D., McGreevy, B. P., & Barraclough, D. J. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.07.003 · Cited by 123 · see citing works
Self-citationFunctional Specialization of the Primate Frontal Cortex during Decision Making: Figure 1.
Lee, D., Rushworth, M. F. S., Walton, M. E., Watanabe, M., & Sakagami, M. (2007) · Journal of Neuroscience · doi:10.1523/jneurosci.1561-07.2007 · Cited by 121 · see citing works
Self-citationDELAYED REWARD AND COST DISCOUNTING
Murphy, J. G., Vuchinich, R. E., & Simpson, C. A. (2001) · OpenSIUC (Southern Illinois University Carbondale) · Cited by 111 · see citing works
Self-control in the monkeymacaca fascicularis
Tobin, H., Logue, A. W., Chelonis, J. J., Ackerman, K. T., & May, J. G. (1996) · Animal Learning & Behavior · doi:10.3758/bf03198964 · Cited by 107 · see citing works
Short-Term Temporal Discounting of Reward Value in Human Ventral Striatum
Gregorios-Pippas, L., Tobler, P. N., & Schultz, W. (2009) · Journal of Neurophysiology · doi:10.1152/jn.90730.2008 · Cited by 101 · see citing works
Neural basis of quasi-rational decision making
Lee, D. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.02.001 · Cited by 98 · see citing works
Self-citationIntangibility in intertemporal choice
Rick, S., & Loewenstein, G. (2008) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0150 · Cited by 95 · see citing works
Goal-directed control and its antipodes
Dayan, P. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.004 · Cited by 93 · see citing works
Reward prediction based on stimulus categorization in primate lateral prefrontal cortex
Pan, X., Sawa, K., Tsuda, I., Tsukada, M., & Sakagami, M. (2008) · Nature Neuroscience · doi:10.1038/nn.2128 · Cited by 84 · see citing works
Delay discounting of cocaine by rhesus monkeys.
Woolverton, W. L., Myerson, J., & Green, L. (2007) · Experimental and Clinical Psychopharmacology · doi:10.1037/1064-1297.15.3.238 · Cited by 82 · see citing works
Cardinal, R. N., Daw, N. D., Robbins, T. W., & Everitt, B. J. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00053-9 · Cited by 70 · see citing works
Roesch, M. R., & Olson, C. R. (2005) · Journal of Neurophysiology · doi:10.1152/jn.00064.2005 · Cited by 69 · see citing works
Cortical mechanisms for reinforcement learning in competitive games
Seo, H., & Lee, D. (2008) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2008.0158 · Cited by 65 · see citing works
Self-citationNeuronal Activity Representing Temporal Prediction of Reward in the Primate Prefrontal Cortex
Tsujimoto, S., & Sawaguchi, T. (2005) · Journal of Neurophysiology · doi:10.1152/jn.01149.2004 · Cited by 64 · see citing works
Brain pathways for cognitive-emotional decision making in the human animal
Levine, D. S. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.003 · Cited by 45 · see citing works
Encoding of Action History in the Rat Ventral Striatum
Kim, Y. B., Huh, N., Lee, H., Baeg, E. H., Lee, D., & Jung, M. W. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00310.2007 · Cited by 42 · see citing works
Self-citationI Was Pleased a Moment Ago: How Pleasure Varies with Background and Foreground Reference Points
Heyman, J., Mellers, B. A., Tishcenko, S., & Schwartz, A. (2004) · Motivation and Emotion · doi:10.1023/b:moem.0000027278.22201.68 · Cited by 25 · see citing works
Adaptive learning via selectionism and Bayesianism, Part I: Connection between the two
Zhang, J. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.018 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Updating dopamine reward signals
Schultz, W. (2013) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.11.012 · Cited by 651 · see citing works
A reservoir of time constants for memory traces in cortical neurons
Bernacchia, A., Seo, H., Lee, D., & Wang, X.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2752 · Cited by 292 · see citing works
“where Rew(t) denotes the reward received by the animal in trial t, and the term inside square is commonly defined as the reward prediction error; that is, the (12)”
BackgroundSelf-citationLateral Intraparietal Cortex and Reinforcement Learning during a Mixed-Strategy Game
Seo, H., Barraclough, D. J., & Lee, D. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1479-09.2009 · Cited by 125 · see citing works
Self-citationEffects of α-2A adrenergic receptor agonist on time and risk preference in primates
Kim, S., Bobeica, I., Gamo, N. J., Arnsten, A. F. T., & Lee, D. (2012) · Psychopharmacology · doi:10.1007/s00213-011-2520-0 · Cited by 47 · see citing works
Self-citation
- Authors
- Kim, S., Hwang, J., Seo, H., & Lee, D.
- Affiliations
- Yale University (US)
- University of Rochester (US)
- Corresponding author
- Lee, Daeyeol
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2009-04
- Volume
- 22
- Issue
- 3
- Pages
- 294-304
- ISSN
- 0893-6080 (ISSN-L)
- PMC
- PMC2693219
- OpenAlex
- W2011389044
- Cited by
- 37
- References
- 82
- Citations by year
- 2024: 2 · 2023: 3 · 2022: 3 · 2021: 2 · 2020: 1 · 2019: 1 · 2017: 3 · 2016: 1 · 2015: 2 · 2014: 1 · 2013: 1 · 2012: 5
- FWCI
- 2.74
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (MH073246, R01MH073246)
- National Institute on Drug Abuse (RL1DA024855, DA024855)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- intertemporal choice, delayed rewards, subjective value, prefrontal cortex, lateral prefrontal cortex, hyperbolic discounting, reward magnitude, reward delay, Nash equilibrium, reinforcement learning, action-outcome associations, decision-making, single-unit activity, rhesus monkeys
- MeSH terms
- Algorithms, Animals, Haplorhini, Association Learning, Decision Making, Game Theory, Learning, Nerve Net, Neurons, Reinforcement, Psychology, Prefrontal Cortex
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
