Pavlovian valuation systems in learning and decision making
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Environmental stimuli guide value-based decision making, but can do so through cognitive representation of outcomes or through general-incentive properties attributed to the cues themselves. We assert that these differences are conferred through the use of alternative associative structures differing in computational intensity. Using this framework, we review scientific evidence to discern the neural substrates of these assumed separable processes. We suggest that the contribution of the mesolimbic dopamine system to Pavlovian valuation is restricted to an affective system that is only updated through experiential feedback of stimulus-outcome pairing, whereas the orbitofrontal cortex contributes to an alternative system capable of inferential reasoning. Finally we discuss the interactions and convergence of these systems and their implications for decision making and its pathology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maps of Bounded Rationality: Psychology for Behavioral Economics
Kahneman, D. (2003) · American Economic Review · doi:10.1257/000282803322655392 · Cited by 5,377
Di Chiara, G., & Imperato, A. (1988) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.85.14.5274 · Cited by 4,733
Dopamine, learning and motivation.
Wise, R. A. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1406 · Cited by 3,379 · cited 1×
“Dopamine and model-free Pavlovian valuation A role for dopamine in reward learning is suggested by data obtained from multiple paradigms [19,20,21 ].”
MethodDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
From motivation to action: Functional interface between the limbic system and the motor system
MOGENSON, G., JONES, D., & YIM, C. (1980) · Progress in Neurobiology · doi:10.1016/0301-0082(80)90018-0 · Cited by 2,331 · cited 1×
“These circuits access common effector systems to influence motor behavior, with the mesolimbic pathway and OFC efferents converging in the ventral striatum (Figure 2), a structure previously identified as a ‘limbic-motor interface’ [72].”
BackgroundA framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184 · cited 2×
“This framework has been applied to the study of habitual and goal-directed instrumental behavior to learn optimal actions [2,3,11].”
BackgroundMethodA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“The similarity between these dopaminergic responses and the reward-prediction errors used by model-free algorithms was articulated by Montague and colleagues [24].”
MethodThe reinstatement model of drug relapse: history, methodology and major findings
Shaham, Y., Shalev, U., Lu, L., de Wit, H., & Stewart, J. (2003) · Psychopharmacology · doi:10.1007/s00213-002-1224-x · Cited by 1,615
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · cited 1×
“Consistent with the hypothesis that the mesolimbic dopamine pathway transmits reward-prediction-error signals, qualitatively similar signals have been extracted from blood-oxygen-level-dependent (BOLD) hemodynamic response in the human ventral striatum, following the presentation of rewards or reward-predictive cues […”
ResultTwo types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378
Neuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318
Relative reward preference in primate orbitofrontal cortex
Tremblay, L., & Schultz, W. (1999) · Nature · doi:10.1038/19525 · Cited by 1,314 · cited 1×
“Subpopulations of OFC neurons in rodents and primates encode a variety of task parameters related to the value and identity of outcomes, both actual and expected [38–45].”
BackgroundMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 1×
“Subsequently, the reward-prediction-error hypothesis of dopamine function has been supported by experimental evidence obtained in many scenarios examining appetitive processing [25–27,28 ], particularly in more medial aspects of dopaminergic midbrain [29 ] that project to limbic forebrain structures.”
ResultPhasic Firing in Dopaminergic Neurons Is Sufficient for Behavioral Conditioning
Tsai, H.-C., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., & Deisseroth, K. (2009) · Science · doi:10.1126/science.1168878 · Cited by 1,266 · cited 1×
“Dopamine and model-free Pavlovian valuation A role for dopamine in reward learning is suggested by data obtained from multiple paradigms [19,20,21 ].”
MethodToward a behavioral economic understanding of drug dependence: delay discounting processes
Bickel, W. K., & Marsch, L. A. (2001) · Addiction · doi:10.1046/j.1360-0443.2001.961736.x · Cited by 1,117
A selective role for dopamine in stimulus-reward learning
Flagel, S. B., Clark, J. J., Robinson, T. E., Mayo, L., Czuj, A., Willuhn, I., Akers, C. A., Clinton, S. M., Phillips, P. E. M., & Akil, H. (2011) · Nature · doi:10.1038/nature09588 · Cited by 1,026
Self-citationDopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018 · cited 1×
“Subsequently, the reward-prediction-error hypothesis of dopamine function has been supported by experimental evidence obtained in many scenarios examining appetitive processing [25–27,28 ], particularly in more medial aspects of dopaminergic midbrain [29 ] that project to limbic forebrain structures.”
ResultResponses of monkey dopamine neurons during learning of behavioral reactions
Ljungberg, T., Apicella, P., & Schultz, W. (1992) · Journal of Neurophysiology · doi:10.1152/jn.1992.67.1.145 · Cited by 987
Volkow, N. D., Fowler, J. S., Wang, G., Hitzemann, R., Logan, J., Schlyer, D. J., Dewey, S. L., & Wolf, A. P. (1993) · Synapse · doi:10.1002/syn.890140210 · Cited by 924
Neural correlates, computation and behavioural impact of decision confidence
Kepecs, A., Uchida, N., Zariwala, H. A., & Mainen, Z. F. (2008) · Nature · doi:10.1038/nature07200 · Cited by 809
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792
Mindfulness-Based Relapse Prevention for Substance Use Disorders: A Pilot Efficacy Trial
Bowen, S., Chawla, N., Collins, S. E., Witkiewitz, K., Hsu, S., Grow, J., Clifasefi, S., Garner, M., Douglass, A., Larimer, M. E., et al. (2009) · Substance Abuse · doi:10.1080/08897070903250084 · Cited by 715 · cited 1×
“Indeed, this practice is used with some success in treating drug addiction [83], presumably because it renders decision making under a biological domain, dominated by the OFC, where values assigned to predictive stimuli are not potentiated by drugs themselves and behaviors are less sensitive to stimulus-driven influen…”
BackgroundAssociative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Day, J. J., Roitman, M. F., Wightman, R. M., & Carelli, R. M. (2007) · Nature Neuroscience · doi:10.1038/nn1923 · Cited by 710
BOLD responses reflecting dopaminergic signals in the human ventral tegmental area
D'Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008) · Science · doi:10.1126/science.1150605 · Cited by 661
Orbitofrontal Cortex and Representation of Incentive Value in Associative Learning
Gallagher, M., McMahan, R. W., & Schoenbaum, G. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-15-06610.1999 · Cited by 593
Corbit, L. H., & Balleine, B. W. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.4507-04.2005 · Cited by 589
Izquierdo, A., Suda, R. K., & Murray, E. A. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.1921-04.2004 · Cited by 588
Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex
Rolls, E., & Baylis, L. (1994) · The Journal of Neuroscience · doi:10.1523/jneurosci.14-09-05437.1994 · Cited by 587
A new perspective on the role of the orbitofrontal cortex in adaptive behaviour
Schoenbaum, G., Roesch, M. R., Stalnaker, T. A., & Takahashi, Y. K. (2009) · Nature reviews. Neuroscience · doi:10.1038/nrn2753 · Cited by 585
A unified framework for addiction: Vulnerabilities in the decision process
Redish, A. D., Jensen, S., & Johnson, A. (2008) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0800472x · Cited by 501
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Critchley, H. D., & Rolls, E. T. (1996) · Journal of Neurophysiology · doi:10.1152/jn.1996.75.4.1673 · Cited by 468
Pan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429
Zweifel, L. S., Parker, J. G., Lobb, C. J., Rainwater, A., Wall, V. Z., Fadok, J. P., Darvas, M., Kim, M. J., Mizumori, S. J. Y., Paladini, C. A., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0813415106 · Cited by 426
Self-citationRobinson, T. E., & Flagel, S. B. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.09.006 · Cited by 416 · cited 1×
“Current Opinion in Neurobiology 2012, 22:1054–1061 response do not exhibit incentive attribution to the CS [36], do not encode reward-prediction-error signals by mesolimbic dopamine [34 ] (Figure 1), and do not require dopamine for learning to occur [34 ].”
BackgroundWalton, M. E., Behrens, T. E. J., Buckley, M. J., Rudebeck, P. H., & Rushworth, M. F. S. (2010) · Neuron · doi:10.1016/j.neuron.2010.02.027 · Cited by 397
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 1×
“Indeed, theoretical work has provided computational accounts for such state representation and its influence on Pavlovian conditioning not generally captured by previous modelfree reinforcement learning algorithms [70,71].”
BackgroundOrbitofrontal Cortex Mediates Outcome Encoding in Pavlovian But Not Instrumental Conditioning
Ostlund, S. B., & Balleine, B. W. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5443-06.2007 · Cited by 379 · cited 1×
“OFC lesions disrupt such learning from changes in outcome identity [51 ,52], interfere with the ability of these cue-evoked outcome expectancies to serve as conditioned reinforcers [52], and disrupt the cues’ ability to selectively enhance responding for the specific outcomes the cues predict [53].”
BackgroundRedish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346
Corbit, L. H., & Balleine, B. W. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2711-11.2011 · Cited by 309
Encoding of Time-Discounted Rewards in Orbitofrontal Cortex Is Independent of Value Representation
Roesch, M. R., Taylor, A. R., & Schoenbaum, G. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.027 · Cited by 297
How visual stimuli activate dopaminergic neurons at short latency
Dommett, E., Coizet, V., Blaha, C. D., Martindale, J., Lefebvre, V., Walton, N., Mayhew, J. E. W., Overton, P. G., & Redgrave, P. (2005) · Science · doi:10.1126/science.1107026 · Cited by 285
Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280 · cited 2×
“Following OFC lesions, dopamine neurons in the VTA remain responsive to reward outcomes and predictive cues, but do not show their typical reward-predictionerror-like modulation during learning or differential encoding of expected reward value [65 ].”
BackgroundDoes the orbitofrontal cortex signal value?
Schoenbaum, G., Takahashi, Y., Liu, T., & McDannald, M. A. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06210.x · Cited by 232 · cited 1×
“com stimuli within a complex associative structure that permits inferences relating stimuli and states that have not been experienced [54,55].”
BackgroundTsuchida, A., Doll, B. B., & Fellows, L. K. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1958-10.2010 · Cited by 188 · cited 1×
“related decision-making tasks with dynamically changing reward contingencies [67,68 ].”
BackgroundSingh, T. (2010) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2010.00126 · Cited by 40 · cited 1×
“In addition, lesions of the ventral striatum disrupt learning from changes in either outcome identity or value [51 ], disrupt signtracking behavior [74], and impair sensitivity to devaluation [75].”
Background
Cited by
A framework for studying emotions across species
Anderson, D. J., & Adolphs, R. (2014) · Cell · doi:10.1016/j.cell.2014.03.003 · Cited by 691
“There is now a large literature from reinforcement learning, applied widely across species, that provides important intersection with the study of emotion (Clark et al, 2012; see Box 3).”
Toward a Wiring Diagram Understanding of Appetite Control
Andermann, M. L., & Lowell, B. B. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.014 · Cited by 606
Dissecting the diversity of midbrain dopamine neurons
Roeper, J. (2013) · Trends in Neurosciences · doi:10.1016/j.tins.2013.03.003 · Cited by 373
Dopamine neurons projecting to the posterior striatum form an anatomically distinct subclass
Menegas, W., Bergan, J. F., Ogawa, S. K., Isogai, Y., Umadevi Venkataraju, K., Osten, P., Uchida, N., & Watabe-Uchida, M. (2015) · eLife · doi:10.7554/elife.10032 · Cited by 322
A Unified Framework for Dopamine Signals across Timescales
Kim, H. R., Malik, A. N., Mikhael, J. G., Bech, P., Tsutsui-Kimura, I., Sun, F., Zhang, Y., Li, Y., Watabe-Uchida, M., Gershman, S. J., et al. (2020) · Cell · doi:10.1016/j.cell.2020.11.013 · Cited by 302
Metadata
- Authors
- Clark, J. J., Hollon, N. G., & Phillips, P. E.
- Venue
- Current Opinion in Neurobiology
- Publisher
- Elsevier BV
- Published
- 2012-12
- Volume
- 22
- Issue
- 6
- Pages
- 1054-1061
- ISSN
- 0959-4388 (ISSN-L)
- OpenAlex
- W2080856558
- Cited by
- 121
- References
- 47
- FWCI
- 0.60
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.