Dopamine neurons learn relative chosen value from probabilistic rewards
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Economic theories posit reward probability as one of the factors defining reward value. Individuals learn the value of cues that predict probabilistic rewards from experienced reward frequencies. Building on the notion that responses of dopamine neurons increase with reward probability and expected value, we asked how dopamine neurons in monkeys acquire this value signal that may represent an economic decision variable. We found in a Pavlovian learning task that reward probability-dependent value signals arose from experienced reward frequencies. We then assessed neuronal response acquisition during choices among probabilistic rewards. Here, dopamine responses became sensitive to the value of both chosen and unchosen options. Both experiments showed also the novelty responses of dopamine neurones that decreased as learning advanced. These results show that dopamine neurons acquire predictive value signals from the frequency of experienced rewards. This flexible and fast signal reflects a specific decision variable and could update neuronal decision mechanisms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · cited 4×
“…according to: atþ1ðxtÞ ¼ h dtj jþ ð1 hÞatðxtÞ (3) where h is free parameter (a constant) which defines the degree to which the learning rate used in the current trial, at should be modified based on the experienced prediction error, dt (Pearce and Hall, 1980; Pearce et al., 1981; Le Pelley, 2004).”
BackgroundMethodInfluentialDopamine in motivational control: rewarding, aversive, and alerting
Bromberg-Martin, E. S., Matsumoto, M., & Hikosaka, O. (2010) · Neuron · doi:10.1016/j.neuron.2010.11.022 · Cited by 2,396 · cited 2×
“Dopamine signals induce value learning (Steinberg et al., 2013) and are implicated in multiple aspects of goal-directed behaviour (Schultz, 1998; Bromberg-Martin et al., 2010; Stauffer et al., 2016).”
BackgroundDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085 · cited 4×
“Cues that predict reward with high probability evoke larger responses than cues predicting the same reward with lower probability (Fiorillo et al., 2003).”
BackgroundResultInfluentialSelf-citationThe formation of learning sets.
Harlow, H. F. (1949) · Psychological Review · doi:10.1037/h0062474 · Cited by 1,959
Natural neural projection dynamics underlying social behavior
Gunaydin, L. A., Grosenick, L., Finkelstein, J. C., Kauvar, I. V., Fenno, L. E., Adhikari, A., Lammel, S., Mirzabekov, J. J., Airan, R. D., Zalocusky, K. A., et al. (2014) · Cell · doi:10.1016/j.cell.2014.05.017 · Cited by 1,389 · cited 3×
“2 s after cue onset most likely reflected the previously observed novelty signals (Ljungberg et al., 1992; Horvitz et al., 1997; Schultz, 1998; Costa et al., 2014; Gunaydin et al., 2014).”
BackgroundInfluentialNeuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318 · cited 2×
“The phasic activity of dopamine neurons encodes reward prediction error (Schultz et al., 1997; Bayer and Glimcher, 2005; Enomoto et al., 2011; Cohen et al., 2012).”
BackgroundMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 1×
“Dopamine signals induce value learning (Steinberg et al., 2013) and are implicated in multiple aspects of goal-directed behaviour (Schultz, 1998; Bromberg-Martin et al., 2010; Stauffer et al., 2016).”
BackgroundAdaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 3×
“Results from previous studies on fully established tasks suggest that the acquired dopamine responses to probabilistic rewards do not code reward probability on its own but rather increase monotonically with the statistical expected value (Fiorillo et al., 2003; Tobler et al., 2005).”
BackgroundResultSelf-citationDichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190 · cited 1×
“These signals are transmitted to the striatum and cortex where they would be capable to update stimulus and action values (Reynolds et al., 2001; Shen et al., 2008).”
BackgroundResponses of monkey dopamine neurons during learning of behavioral reactions
Ljungberg, T., Apicella, P., & Schultz, W. (1992) · Journal of Neurophysiology · doi:10.1152/jn.1992.67.1.145 · Cited by 987
Self-citationA cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879 · cited 1×
“These signals are transmitted to the striatum and cortex where they would be capable to update stimulus and action values (Reynolds et al., 2001; Shen et al., 2008).”
BackgroundNeurobiology of economic choice: a good-based model
Padoa-Schioppa, C. (2011) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-061010-113648 · Cited by 600
Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597
Importance of unpredictability for reward responses in primate dopamine neurons
Mirenowicz, J., & Schultz, W. (1994) · Journal of Neurophysiology · doi:10.1152/jn.1994.72.2.1024 · Cited by 569
Self-citationMidbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · cited 5×
“Previous studies of dopamine activity during choice have shown chosen value coding by dopamine signals (Morris et al., 2006; Saddoris et al., 2015), but other studies have shown coding of the best available option, irrespective of choice (Roesch et al.”
BackgroundInfluentialPan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429 · cited 2×
“This neuronal acquisition happens gradually (Hollerman and Schultz, 1998), and is well-approximated by reinforcement learning (RL) models (Pan et al., 2005).”
BackgroundZweifel, L. S., Parker, J. G., Lobb, C. J., Rainwater, A., Wall, V. Z., Fadok, J. P., Darvas, M., Kim, M. J., Mizumori, S. J. Y., Paladini, C. A., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0813415106 · Cited by 426 · cited 2×
“…and flexible dopamine responses we observed during choice correspond well to recent findings demonstrating the encoding of economic utility by dopamine neurons (Lak et al., 2014; Stauffer et al., 2014) and the necessity of phasic dopamine responses for consistent choices (Zweifel et al., 2009).”
BackgroundResultLe Pelley, M. E. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000141 · Cited by 360
Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat
Horvitz, J. C., Stewart, T., & Jacobs, B. L. (1997) · Brain Research · doi:10.1016/s0006-8993(97)00265-5 · Cited by 283 · cited 3×
“2 s after cue onset most likely reflected the previously observed novelty signals (Ljungberg et al., 1992; Horvitz et al., 1997; Schultz, 1998; Costa et al., 2014; Gunaydin et al., 2014).”
BackgroundInfluentialDopamine modulates novelty seeking behavior during decision making.
Costa, V. D., Tran, V. L., Turchi, J., & Averbeck, B. B. (2014) · Behavioral Neuroscience · doi:10.1037/a0037128 · Cited by 263 · cited 3×
“2 s after cue onset most likely reflected the previously observed novelty signals (Ljungberg et al., 1992; Horvitz et al., 1997; Schultz, 1998; Costa et al., 2014; Gunaydin et al., 2014).”
BackgroundResultInfluentialDopamine prediction error responses integrate subjective value from different reward dimensions
Lak, A., Stauffer, W. R., & Schultz, W. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1321596111 · Cited by 257 · cited 2×
“…and flexible dopamine responses we observed during choice correspond well to recent findings demonstrating the encoding of economic utility by dopamine neurons (Lak et al., 2014; Stauffer et al., 2014) and the necessity of phasic dopamine responses for consistent choices (Zweifel et al., 2009).”
ResultSelf-citationDopamine Reward Prediction Error Responses Reflect Marginal Utility
Stauffer, W. R., Lak, A., & Schultz, W. (2014) · Current Biology · doi:10.1016/j.cub.2014.08.064 · Cited by 204
Self-citationTemporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli
Nomoto, K., Schultz, W., Watanabe, T., & Sakagami, M. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4828-09.2010 · Cited by 158 · cited 2×
“During choice, the acquired dopamine responses reflected the value of the chosen option relative to the unchosen option.”
BackgroundResultSelf-citationDopamine neurons learn to encode the long-term value of multiple future rewards
Enomoto, K., Matsumoto, N., Nakai, S., Satoh, T., Sato, T. K., Ueda, Y., Inokawa, H., Haruno, M., & Kimura, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014457108 · Cited by 83
Scaling prediction errors to reward variability benefits error-driven learning in humans
Diederen, K. M. J., & Schultz, W. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00483.2015 · Cited by 69 · cited 1×
“First, the development of neuronal responses over trials as well as animals’ choices were best explained by models that adaptively adjusted their learning rate based on past prediction errors, resembling previous studies in human subjects (Nassar et al., 2010; Diederen and Schultz, 2015).”
ResultSelf-citation
Cited by
Dopamine and Cognitive Control in Prefrontal Cortex
Ott, T., & Nieder, A. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.12.006 · Cited by 530
“Theyarerapidly activatedby primaryreinforcers that areeither rewarding orpunishing,or salient and novel stimuli [49 – 52] (Figure 2B, bottom).”
BackgroundImpaired reward prediction error encoding and striatal-midbrain connectivity in depression
Kumar, P., Goer, F., Murray, L., Dillon, D. G., Beltzer, M. L., Cohen, A. L., Brooks, N. H., & Pizzagalli, D. A. (2018) · Neuropsychopharmacology · doi:10.1038/s41386-018-0032-x · Cited by 219
Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170
Self-citationLearning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity
Coddington, L. T., & Dudman, J. T. (2019) · Neuron · doi:10.1016/j.neuron.2019.08.036 · Cited by 150
Curiosity: primate neural circuits for novelty and information seeking
Monosov, I. E. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00784-9 · Cited by 44
“…the value was conveyed by novel objects at the start of learning than when it was conveyed by the now-familiar objects at later stages of learning 80 . brain areas in the primate brain (displayed on the y -axis) were found to contain neurons that discriminate between novel and familiar objects…”
Metadata
- Authors
- Lak, A., Stauffer, W. R., & Schultz, W.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2016-10-27
- Volume
- 5
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W2544311518
- Cited by
- 91
- References
- 25
- FWCI
- 5.21
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.