Dopamine Reward Prediction Error Responses Reflect Marginal Utility
Open accessAbstract
BACKGROUND: Optimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity. RESULTS: In two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions' shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles. CONCLUSIONS: These data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reward prediction-error response should be specifically considered to be a utility prediction error signal.
“Furthermore, I suggest that the reward prediction-error response should be specifically considered to be a utility prediction error signal35.”
From Dopamine reward prediction-error signalling: A two-component response (2016) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,700 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Theory of games and economic behavior
Garcia, S. R., & Miller, S. J. (2019) · American Mathematical Society eBooks · doi:10.1090/mbk/121/32 · Cited by 16,777 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Self-citation“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891 · see citing works
Risk Aversion and Incentive Effects
Holt, C. A., & Laury, S. K. (2002) · American Economic Review · doi:10.1257/000282802762024700 · Cited by 5,829 · see citing works
The Theory of Games and Economic Behavior
Harrison, R. W., von Neumann, J., & Morgenstern, O. (1945) · Journal of Farm Economics · doi:10.2307/1232672 · Cited by 5,258 · see citing works
Allais, M. (1953) · Econometrica · doi:10.2307/1907921 · Cited by 4,170 · see citing works
Increasing risk: I. A definition
Rothschild, M., & Stiglitz, J. (1970) · Journal of Economic Theory · doi:10.1016/0022-0531(70)90038-4 · Cited by 3,760 · see citing works
The Utility Analysis of Choices Involving Risk
Friedman, M., & Savage, L. J. (1948) · Journal of Political Economy · doi:10.1086/256692 · Cited by 3,128 · see citing works
Exposition of a New Theory on the Measurement of Risk
Bernoulli, D. (1954) · Econometrica · doi:10.2307/1909829 · Cited by 2,779 · see citing works
A Model of Reference-Dependent Preferences
Koszegi, B., & Rabin, M. (2006) · The Quarterly Journal of Economics · doi:10.1093/qje/121.4.1133 · Cited by 2,728 · see citing works
Discrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085 · see citing works
Self-citationThe neural correlates of subjective value during intertemporal choice
Kable, J. W., & Glimcher, P. W. (2007) · Nature Neuroscience · doi:10.1038/nn2007 · Cited by 1,916 · see citing works
Markowitz, H. M. (1952) · Journal of Political Economy · doi:10.1086/257177 · Cited by 1,821 · see citing works
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721 · see citing works
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598 · see citing works
The Expected Utility Model: Its Variants, Purposes, Evidence and Limitations
Schoemaker, P. J. H. (2007) · Journal of Economic Literature · Cited by 1,427 · see citing works
A Theory of Disappointment Aversion
Gül, F. (1991) · Econometrica · doi:10.2307/2938223 · Cited by 1,405 · see citing works
On the Shape of the Probability Weighting Function
Gonzalez, R., & Wu, G. (1999) · Cognitive Psychology · doi:10.1006/cogp.1998.0710 · Cited by 1,393 · see citing works
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Adaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · see citing works
Self-citationMotivational control of goal-directed action
Dickinson, A., & Balleine, B. (1994) · Animal Learning & Behavior · doi:10.3758/bf03199951 · Cited by 995 · see citing works
Choice Under Uncertainty: Problems Solved and Unsolved
Machina, M. J. (1992) · Hübner international series on risk, insurance, and economic security · doi:10.1007/978-94-015-7957-5_2 · Cited by 950 · see citing works
A causal link between prediction errors, dopamine neurons and learning
Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K. A., & Janak, P. H. (2013) · Nature Neuroscience · doi:10.1038/nn.3413 · Cited by 879 · see citing works
Choice Under Uncertainty: Problems Solved and Unsolved
Machina, M. J. (1987) · The Journal of Economic Perspectives · doi:10.1257/jep.1.1.121 · Cited by 878 · see citing works
An empirical demonstration of risk-sensitive foraging preferences
Caraco, T. B., Martindale, S., & Whittam, T. S. (1980) · Animal Behaviour · doi:10.1016/s0003-3472(80)80142-4 · Cited by 767 · see citing works
Decision Making Over Time and Under Uncertainty: A Common Approach
Prelec, D., & Loewenstein, G. F. (1991) · Management Science · doi:10.1287/mnsc.37.7.770 · Cited by 611 · see citing works
Neural Representation of Subjective Value Under Risk and Ambiguity
Levy, I., Snell, J., Nelson, A. J., Rustichini, A., & Glimcher, P. W. (2009) · Journal of Neurophysiology · doi:10.1152/jn.00853.2009 · Cited by 491 · see citing works
Influence of Reward Delays on Responses of Dopamine Neurons
Kobayashi, S., & Schultz, W. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1600-08.2008 · Cited by 431 · see citing works
Self-citationValue Representations in the Primate Striatum during Matching Behavior
Lau, B., & Glimcher, P. W. (2008) · Neuron · doi:10.1016/j.neuron.2008.02.021 · Cited by 411 · see citing works
Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value
Tai, L.-H., Lee, A. M., Benavidez, N., Bonci, A., & Wilbrecht, L. (2012) · Nature Neuroscience · doi:10.1038/nn.3188 · Cited by 399 · see citing works
Utility of Gains and Losses: Measurement-Theoretical and Experimental Approaches
Luce, R. D. (2000) · Medical Entomology and Zoology · Cited by 396 · see citing works
Risk-sensitive neurons in macaque posterior cingulate cortex
McCoy, A. N., & Platt, M. L. (2005) · Nature Neuroscience · doi:10.1038/nn1523 · Cited by 380 · see citing works
Levy, D. J., & Glimcher, P. W. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.2218-11.2011 · Cited by 350 · see citing works
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276 · see citing works
Coding of Reward Risk by Orbitofrontal Neurons Is Mostly Distinct from Coding of Reward Value
O’Neill, M. J., & Schultz, W. (2010) · Neuron · doi:10.1016/j.neuron.2010.09.031 · Cited by 259 · see citing works
Self-citationDopamine prediction error responses integrate subjective value from different reward dimensions
Lak, A., Stauffer, W. R., & Schultz, W. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1321596111 · Cited by 257 · see citing works
Self-citationPrefrontal Coding of Temporally Discounted Values during Intertemporal Choice
Kim, S., Hwang, J., & Lee, D. (2008) · Neuron · doi:10.1016/j.neuron.2008.05.010 · Cited by 241 · see citing works
Encoding of Marginal Utility across Time in the Human Brain
Pine, A., Seymour, B., Roiser, J. P., Bossaerts, P. L., Friston, K. J., Curran, H. V., & Dolan, R. J. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1126-09.2009 · Cited by 201 · see citing works
Risk-dependent reward value signal in human prefrontal cortex
Tobler, P. N., Christopoulos, G. I., O’Doherty, J. P., Dolan, R. J., & Schultz, W. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0809599106 · Cited by 180 · see citing works
Self-citationConvex stochastic dominance with continuous distribution functions
Fishburn, P. C. (1974) · Journal of Economic Theory · doi:10.1016/0022-0531(74)90103-3 · Cited by 138 · see citing works
Thirst-dependent risk preferences in monkeys identify a primitive form of wealth
Yamada, H., Tymula, A. A., Louie, K., & Glimcher, P. W. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1308718110 · Cited by 92 · see citing works
Learning-Related Human Brain Activations Reflecting Individual Finances
Tobler, P. N., Fletcher, P. C., Bullmore, E. T., & Schultz, W. (2007) · Neuron · doi:10.1016/j.neuron.2007.03.004 · Cited by 85 · see citing works
Self-citationPrediction of economic choice by primate amygdala neurons
Fabian Grabenhorst, B., Hernádi, I., & Schultz, W. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212706109 · Cited by 78 · see citing works
Self-citationMonosov, I. E., & Hikosaka, O. (2013) · Nature Neuroscience · doi:10.1038/nn.3398 · Cited by 77 · see citing works
Cardinal Utility for Even-Chance Mixtures of Pairs of Sure Prospects
Debreu, G. (1959) · The Review of Economic Studies · doi:10.2307/2295745 · Cited by 60 · see citing works
Supplementary Eye Field Encodes Reward Prediction Error
So, N., & Stuphorn, V. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.4419-11.2012 · Cited by 56 · see citing works
Effects of α-2A adrenergic receptor agonist on time and risk preference in primates
Kim, S., Bobeica, I., Gamo, N. J., Arnsten, A. F. T., & Lee, D. (2012) · Psychopharmacology · doi:10.1007/s00213-011-2520-0 · Cited by 47 · see citing works
Risk Prediction Error Coding in Orbitofrontal Neurons
O’Neill, M. J., & Schultz, W. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.4236-12.2013 · Cited by 33 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [2]Mas-Colell A. Whinston M.D. Green J.R. Microeconomic Theory 1995 Oxford University Press Oxford
- [5]Perloff J.M. Microeconomics Fifth Edition 2009 Pearson Boston
Cited by
Sorted by citations
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126 · see citing works
Self-citationActive inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156 · see citing works
Dopamine reward prediction-error signalling: A two-component response
Schultz, W. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2015.26 · Cited by 1,036 · see citing works
Self-citationDopamine reward prediction error coding
Schultz, W. (2016) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2016.18.1/wschultz · Cited by 597 · see citing works
Deeply Felt Affect: The Emergence of Valence in Deep Active Inference
Hesp, C., Smith, R., Parr, T., Allen, M., Friston, K. J., & Ramstead, M. J. D. (2021) · Neural Computation · doi:10.1162/neco_a_01341 · Cited by 246 · see citing works
Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170 · see citing works
Self-citationDopamine neurons respond to errors in the prediction of sensory features of expected rewards
Takahashi, Y. K., Batchelor, H. M., Liu, B., Khanna, A., Morales, M., & Schoenbaum, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.025 · Cited by 200 · see citing works
Belief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101 · see citing works
Phasic dopamine signals: from subjective reward value to formal economic utility
Schultz, W., Carelli, R. M., & Wightman, R. M. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.09.006 · Cited by 88 · see citing works
- Authors
- Stauffer, W. R., Lak, A., & Schultz, W.
- Affiliations
- University of Cambridge (GB)
- Corresponding author
- Stauffer, William R.
- Venue
- Current Biology
- Publisher
- Elsevier BV
- Published
- 2014-11
- Volume
- 24
- Issue
- 21
- Pages
- 2491-2500
- ISSN
- 0960-9822 (ISSN-L)
- PMC
- PMC4228052
- OpenAlex
- W1981982902
- Cited by
- 204
- References
- 52
- Citations by year
- 2026: 10 · 2025: 13 · 2024: 10 · 2023: 16 · 2022: 10 · 2021: 26 · 2020: 14 · 2019: 36 · 2018: 19 · 2017: 18 · 2016: 20 · 2015: 11 · 2014: 1
- FWCI
- 9.30
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Current Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Wellcome Trust (095495)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- dopamine, reward prediction error, marginal utility, utility functions, risk preferences, choice under risk, stochastic dominance, certainty equivalent, reinforcement learning, economic decision-making
- MeSH terms
- Animals, Choice Behavior, Dopamine, Macaca mulatta, Male, Probability, Reward, Risk-Taking, Stochastic Processes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
