Neural mechanisms of acquired phasic dopamine responses in learning
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
What biological mechanisms underlie the reward-predictive firing properties of midbrain dopaminergic neurons, and how do they relate to the complex constellation of empirical findings understood as Pavlovian and instrumental conditioning? We previously presented PVLV, a biologically-inspired Pavlovian learning algorithm accounting for DA activity in terms of two interrelated systems: a primary value (PV) system, which governs how DA cells respond to a US (reward) and; a learned value (LV) system, which governs how DA cells respond to a CS. Here, we provide a more extensive review of the biological mechanisms supporting phasic DA firing and their relation to the spate of Pavlovian conditioning phenomena and their sensitivity to focal brain lesions. We further extend the model by incorporating a new NV (novelty value) component reflecting the ability of novel stimuli to trigger phasic DA firing, providing “novelty bonuses” which encourages exploratory working memory updating and in turn speeds learning in trace conditioning and other working memory-dependent paradigms. The evolving PVLV model builds upon insights developed in many earlier computational models, especially reinforcement learning models based on the ideas of Sutton and Barto, biological models, and the psychological model developed by Savastano and Miller. The PVLV framework synthesizes these various approaches, overcoming important shortcomings of each by providing a coherent and specific mapping to much of the relevant empirical data at both the micro- and macro-levels, and examines their relevance for higher order cognitive functions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 8×
“As noted, the computationally oriented literature regarding DA’s role in Pavlovian conditioning has heretofore been dominated by the elegant temporal differences (TD) algorithm of Sutton and Barto (Sutton, 1988; Sutton and Barto, 1990, 1998), although there have also been a few non-TD based accounts put forward (e.g.,…”
BackgroundInfluentialPredictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 3×
“…al., 1990, 1991; Huang and Kandel, 1995; Wickens et al., 1996; Chong et al., 2006), there now exists a broad consensus that phasic DA firing is a learning signal for both Pavlovian and instrumental conditioning (e.g., Montague et al., 1997; Schultz, 1998; Waelti et al., 2001; Wickens et al., 2007).”
BackgroundThe hippocampal-VTA loop: controlling the entry of information into long-term memory
Lisman, J. E., & Grace, A. A. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.002 · Cited by 1,983
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Self-citationThe Emotional Brain, Fear, and the Amygdala
LeDoux, J. (2003) · Cellular and Molecular Neurobiology · doi:10.1023/a:1025048802629 · Cited by 1,818
Neuronal Coding of Prediction Errors
Schultz, W., & Dickinson, A. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.473 · Cited by 1,427 · cited 4×
“…reward prediction error signal first proposed by Rescorla and Wagner (1972) and since widely adopted as the dominant account of Pavlovian conditioning (e.g., Montague et al., 1997; Schultz et al., 1997; Schultz, 1998; Hollerman and Schultz, 1998; Schultz and Dickinson, 2000; Waelti et al., 2001).”
BackgroundInfluentialMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 1×
“…has also been described for other categories of salient stimuli besides positivelyvalenced reward (e.g., aversive, novel, or high intensity stimuli; Bayer and Glimcher, 2005; Bayer et al., 2007; Horvitz et al., 1997; Ljungberg et al., 1992; Steinfels et al., 1983; Legault and Wise, 2001; Horvitz,…”
BackgroundLateral habenula as a source of negative reward signals in dopamine neurons
Matsumoto, M., & Hikosaka, O. (2007) · Nature · doi:10.1038/nature05860 · Cited by 1,292 · cited 4×
“…habenular nucleus of the epithalamus can produce pauses in DA cell firing (e.g., Heldt and Ressler, 2007; Ji and Shepard, 2007; Lecourtier et al., 2008; Matsumoto and Hikosaka, 2007; Shepard et al., 2006) suggest that it may be important for generating the dip associated with omitted rewards.”
BackgroundInfluential
Cited by
The neurobiology of anhedonia and other reward-related deficits
Der-Avakian, A., & Markou, A. (2012) · Trends in Neurosciences · doi:10.1016/j.tins.2011.11.005 · Cited by 1,130
“Hence, ventral striatal dopamine regulates the prediction and anticipation of rewards, two mechanisms responsible for basic reinforcement learning (reviewed in [61]).”
BackgroundAshok, A. H., Marques, T. R., Jauhar, S., Nour, M. M., Goodwin, G. M., Young, A. H., & Howes, O. D. (2017) · Molecular Psychiatry · doi:10.1038/mp.2017.16 · Cited by 546
A high-resolution probabilistic in vivo atlas of human subcortical brain nuclei
Pauli, W. M., Nili, A. N., & Tyszka, J. M. (2018) · Scientific Data · doi:10.1038/sdata.2018.63 · Cited by 517
Neural Circuitry of Reward Prediction Error
Watabe-Uchida, M., Eshel, N., & Uchida, N. (2017) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-072116-031109 · Cited by 437
Collins, A. G. E., & Frank, M. J. (2014) · Psychological Review · doi:10.1037/a0037015 · Cited by 403
Self-citation
Metadata
- Authors
- Hazy, T. E., Frank, M. J., & O’Reilly, R. C.
- Venue
- Neuroscience & Biobehavioral Reviews
- Publisher
- Elsevier BV
- Published
- 2010-04
- Volume
- 34
- Issue
- 5
- Pages
- 701-720
- ISSN
- 0149-7634 (ISSN-L)
- OpenAlex
- W2097649904
- Cited by
- 102
- References
- 53
- FWCI
- 3.14
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Neural mechanisms of acquired phasic dopamine responses in learning. https://app.cochat.ai/papers/10.1016/j.neubiorev.2009.11.019
- The paper
- Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2010). Neural mechanisms of acquired phasic dopamine responses in learning. Neuroscience & Biobehavioral Reviews. https://doi.org/10.1016/j.neubiorev.2009.11.019
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.