Dopaminergic and Prefrontal Basis of Learning from Sensory Confidence and Reward Value
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Deciding between stimuli requires combining their learned value with one's sensory confidence. We trained mice in a visual task that probes this combination. Mouse choices reflected not only present confidence and past rewards but also past confidence. Their behavior conformed to a model that combines signal detection with reinforcement learning. In the model, the predicted value of the chosen option is the product of sensory confidence and learned value. We found precise correlates of this variable in the pre-outcome activity of midbrain dopamine neurons and of medial prefrontal cortical neurons. However, only the latter played a causal role: inactivating medial prefrontal cortex before outcome strengthened learning from the outcome. Dopamine neurons played a causal role only after outcome, when they encoded reward prediction errors graded by confidence, influencing subsequent choices. These results reveal neural signals that combine reward value with sensory confidence and guide subsequent learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Neural Basis of Decision Making
Gold, J. I., & Shadlen, M. N. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.113038 · Cited by 3,764 · cited 1×
“, the probability that a percept is correct (Gold and Shadlen, 2007; Kepecs et al., 2008; Kiani and Shadlen, 2009).”
Top-Down Versus Bottom-Up Control of Attention in the Prefrontal and Posterior Parietal Cortices
Buschman, T. J., & Miller, E. K. (2007) · Science · doi:10.1126/science.1138071 · Cited by 2,380 · cited 1×
“708 Neuron 105, 700–711, February 19, 2020 methods is that we manipulated mPFC with laser pulses at 25 Hz, which could have led to beta frequency oscillations, and might have functional consequences (Buschman and Miller, 2007; van de Vijver et al., 2011).”
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598 · cited 2×
“For example, signals encoding economic chosen value have been found in other prefrontal areas (Padoa-Schioppa and Assad, 2006).”
Natural neural projection dynamics underlying social behavior
Gunaydin, L. A., Grosenick, L., Finkelstein, J. C., Kauvar, I. V., Fenno, L. E., Adhikari, A., Lammel, S., Mirzabekov, J. J., Airan, R. D., Zalocusky, K. A., et al. (2014) · Cell · doi:10.1016/j.cell.2014.05.017 · Cited by 1,389 · cited 1×
“To measure the activity of dopamine neurons, we employed fiber photometry (Gunaydin et al., 2014; Lerner et al., 2015).”
MethodNeuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 2×
“These neurons encode predicted value prior to outcome and reward prediction error after outcome (Bayer and Glimcher, 2005; Cohen et al., 2012; Schultz et al., 1997).”
BackgroundResultPhasic Firing in Dopaminergic Neurons Is Sufficient for Behavioral Conditioning
Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., & Deisseroth, K. (2009) · Science · doi:10.1126/science.1168878 · Cited by 1,266 · cited 1×
“They play a causal role in learning from past rewards (Hamid et al., 2016; Kim et al., 2012; Parker et al., 2016; Stauffer et al., 2016; Steinberg et al., 2013; Tsai et al., 2009), and their responses are graded not only by reward value but also by sensory confidence (Lak et al.”
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126 · cited 2×
“Consistent with previous reports, we found that VTA dopamine responses to outcomes reflect reward size and correlate with future choices (Bayer and Glimcher, 2005; Schultz, 2015).”
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
Ventral tegmental area: cellular heterogeneity, connectivity and behaviour
Morales, M., & Margolis, E. B. (2017) · Nature reviews. Neuroscience · doi:10.1038/nrn.2016.165 · Cited by 1,108
Circuit architecture of VTA dopamine neurons revealed by systematic input-output mapping
Beier, K. T., Steinberg, E. E., DeLoach, K. E., Xie, S., Miyamichi, K., Schwarz, L., Gao, X. J., Kremer, E. J., Malenka, R. C., & Luo, L. (2015) · Cell · doi:10.1016/j.cell.2015.07.015 · Cited by 1,068
Representation of Confidence Associated with a Decision by Neurons in the Parietal Cortex
Kiani, R., & Shadlen, M. N. (2009) · Science · doi:10.1126/science.1169405 · Cited by 1,027 · cited 2×
“Signals encoding sensory confidence (one of the two factors that determine predicted value) have been seen in parietal cortex, orbitofrontal cortex, and dorsal pulvinar (Kepecs et al., 2008; Kiani and Shadlen, 2009; Komura et al., 2013), and signals necessary for learning may be observed in orbitofrontal cortex (Mille…”
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · cited 2×
“Studies of reward learning revealed how decisions are informed by past rewards and modeled this process as reinforcement learning (Daw and Doya, 2006; Lee et al., 2012; Samejima et al., 2005; Schultz, 2015; Sutton and Barto, 1998).”
Neural correlates, computation and behavioural impact of decision confidence
Kepecs, A., Uchida, N., Zariwala, H. A., & Mainen, Z. F. (2008) · Nature · doi:10.1038/nature07200 · Cited by 809 · cited 2×
“Signals encoding sensory confidence (one of the two factors that determine predicted value) have been seen in parietal cortex, orbitofrontal cortex, and dorsal pulvinar (Kepecs et al., 2008; Kiani and Shadlen, 2009; Komura et al., 2013), and signals necessary for learning may be observed in orbitofrontal cortex (Mille…”
Intact-brain analyses reveal distinct information carried by SNc dopamine subcircuits
Lerner, T. N., Shilyansky, C., Davidson, T. J., Evans, K. E., Beier, K. T., Zalocusky, K. A., Crow, A. K., Malenka, R. C., Luo, L., Tomer, R., et al. (2015) · Cell · doi:10.1016/j.cell.2015.07.014 · Cited by 794
Coordination of Actions and Habits in the Medial Prefrontal Cortex of Rats
Killcross, S. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.4.400 · Cited by 748
Flow of cortical activity underlying a tactile decision in mice
Guo, Z. V., Li, N., Huber, D., Ophir, E., Gutnisky, D., Ting, J. T., Feng, G., & Svoboda, K. (2014) · Neuron · doi:10.1016/j.neuron.2013.10.020 · Cited by 715 · cited 2×
“We tested these predictions through optogenetic inactivation inmice expressing Channelrhodopsin-2 (Chr2) inPvalb-expressing inhibitory neurons of mPFC (Figures 4A and 4B; Figure S4A) (Guo et al., 2014; Olsen et al., 2012).”
Hippocampal–prefrontal input supports spatial encoding in working memory
Spellman, T., Rigotti, M., Ahmari, S. E., Fusi, S., Gogos, J. A., & Gordon, J. A. (2015) · Nature · doi:10.1038/nature14445 · Cited by 693 · cited 1×
“Studies in animals freely moving in a maze or in an operant box argue that PFC neurons can be selective for directional actions (e.g., Feierstein et al., 2006; Spellman et al., 2015; Sul et al., 2010).”
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Gain control by layer six in cortical circuits of vision
Olsen, S. R., Bortone, D. S., Adesnik, H., & Scanziani, M. (2012) · Nature · doi:10.1038/nature10835 · Cited by 539 · cited 2×
“We tested these predictions through optogenetic inactivation inmice expressing Channelrhodopsin-2 (Chr2) inPvalb-expressing inhibitory neurons of mPFC (Figures 4A and 4B; Figure S4A) (Guo et al., 2014; Olsen et al., 2012).”
Choosing the greater of two goods: neural currencies for valuation and decision making
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1666 · Cited by 536
A 4 Hz oscillation adaptively synchronizes prefrontal, VTA, and hippocampal activities
Fujisawa, S., & Buzsáki, G. (2011) · Neuron · doi:10.1016/j.neuron.2011.08.018 · Cited by 501 · cited 2×
“…responses would be consistent with anatomical projections (Beier et al., 2015; Carr and Sesack, 2000; Morales and Margolis, 2017), with simultaneous frontal-VTA recordings (Fujisawa and Buzsáki, 2011), and with recordings following pharmacological manipulations (Starkweather et al., 2018).”
BackgroundThe role of prelimbic cortex in instrumental conditioning
Corbit, L. H., & Balleine, B. W. (2003) · Behavioural Brain Research · doi:10.1016/j.bbr.2003.09.023 · Cited by 423
Prefrontal cortex output circuits guide reward seeking through divergent cue encoding
Otis, J. M., Namboodiri, V. M. K., Matan, A. M., Voets, E. S., Mohorn, E. P., Kosyk, O., McHenry, J. A., Robinson, J. E., Resendez, S. L., Rossi, M. A., et al. (2017) · Nature · doi:10.1038/nature21376 · Cited by 412 · cited 2×
“Neurons in mPFC encode future rewards as inferred from past outcomes (Moorman and Aston-Jones, 2015; Otis et al., 2017; Pinto and Dan, 2015; Pratt and Mizumori, 2001; Rushworth et al., 2011).”
Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value
Tai, L.-H., Lee, A. M., Benavidez, N., Bonci, A., & Wilbrecht, L. (2012) · Nature Neuroscience · doi:10.1038/nn.3188 · Cited by 399 · cited 1×
“First, where are the learned values stored? Previous studies suggest striatal neurons and neurons in frontal cortex are plausible candidates, because they encode the learned value of stimuli and actions and they receive strong dopamine projections (Ding and Gold, 2013; Lee et al., 2012; Padoa-Schioppa and Assad, 2006;…”
Cell-Type-Specific Activity in Prefrontal Cortex during Goal-Directed Behavior
Pinto, L., & Dan, Y. (2015) · Neuron · doi:10.1016/j.neuron.2015.06.021 · Cited by 323 · cited 1×
“Neurons in mPFC encode future rewards as inferred from past outcomes (Moorman and Aston-Jones, 2015; Otis et al., 2017; Pinto and Dan, 2015; Pratt and Mizumori, 2001; Rushworth et al., 2011).”
A computational framework for the study of confidence in humans and animals
Kepecs, A., & Mainen, Z. F. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0037 · Cited by 305
Ostlund, S. B., & Balleine, B. W. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1921-05.2005 · Cited by 291
Responses of pulvinar neurons reflect a subject's confidence in visual categorization
Komura, Y., Nikkuni, A., Hirashima, N., Uetake, T., & Miyamoto, A. (2013) · Nature Neuroscience · doi:10.1038/nn.3393 · Cited by 226
Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170
Self-citationSummerfield, C., & Tsetsos, K. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00070 · Cited by 167
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · cited 1×
“, 2015; Carr and Sesack, 2000; Morales and Margolis, 2017), with simultaneous frontal-VTA recordings (Fujisawa and Buzsáki, 2011), and with recordings following pharmacological manipulations (Starkweather et al., 2018).”
Functional Coupling between the Prefrontal Cortex and Dopamine Neurons in the Ventral Tegmental Area
Gao, M., Liu, C.-L., Yang, S., Jin, G.-Z., Bunney, B. S., & Shi, W.-X. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5347-06.2007 · Cited by 125 · cited 1×
“predicted value from observed reward (Eshel et al., 2015; Gao et al., 2007).”
Cited by
The mouse prefrontal cortex: Unity in diversity
Le Merre, P., Ährlund-Richter, S., & Carlén, M. (2021) · Neuron · doi:10.1016/j.neuron.2021.03.035 · Cited by 207
Behavior- and Modality-General Representation of Confidence in Orbitofrontal Cortex
Masset, P., Ott, T., Lak, A., Hirokawa, J., & Kepecs, A. (2020) · Cell · doi:10.1016/j.cell.2020.05.022 · Cited by 109
“Reinforcement learning models that incorporate perceptual ambiguity predict similar choice strategy updating driven by confi-dence-scaled reward prediction errors (Drugowitsch et al., 2019; Lak et al., 2017, Lak et al., 2020b).”
BackgroundMethodInfluentialSelf-citationNeural circuitry of information seeking
Bromberg-Martin, E. S., & Monosov, I. E. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.07.006 · Cited by 89
Nigrostriatal dopamine pathway regulates auditory discrimination behavior
Chen, A. P. F., Malgady, J. M., Chen, L., Shi, K. W., Cheng, E., Plotkin, J. L., Ge, S., & Xiong, Q. (2022) · Nature Communications · doi:10.1038/s41467-022-33747-2 · Cited by 46
Metadata
- Authors
- Lak, A., Okun, M., Moss, M. M., Gurnani, H., Farrell, K., Wells, M. J., Reddy, C. B., Kepecs, A., Harris, K. D., & Carandini, M.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2020-02
- Volume
- 105
- Issue
- 4
- Pages
- 700-711.e6
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2995488106
- Cited by
- 143
- References
- 33
- FWCI
- 6.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.