The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.
Abstract
The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
The debate over dopamine’s role in reward: the case for incentive salience
Berridge, K. C. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0578-x · Cited by 2,347
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Self-citation(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry
Kravitz, A. V., Freeze, B. S., Parker, P. R. L., Kay, K., Thwin, M. T., Deisseroth, K., & Kreitzer, A. C. (2010) · Nature · doi:10.1038/nature09159 · Cited by 1,885
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304
Surmeier, D. J., Ding, J., Day, M., Wang, Z., & Shen, W. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.008 · Cited by 1,203
Dichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190
Tonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · cited 2×
“Broadly, this distinction accords with the separate effects of phasic dopamine encoding prediction errors (Montague et al., 1996), compared with the tonic (baseline) effects of dopamine relating to vigor (Niv et al., 2007).”
BackgroundDistinct roles for direct and indirect pathway striatal neurons in reinforcement
Kravitz, A. V., Tye, L. D., & Kreitzer, A. C. (2012) · Nature Neuroscience · doi:10.1038/nn.3100 · Cited by 972
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Frank, M. J. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929052880093 · Cited by 925
Self-citationA cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879
From reinforcement learning models to psychiatric and neurological disorders
Maia, T. V., & Frank, M. J. (2011) · Nature Neuroscience · doi:10.1038/nn.2723 · Cited by 722
Self-citationFrom prediction error to incentive salience: mesolimbic computation of reward motivation.
Berridge, K. C. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.07990.x · Cited by 700
Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning
Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T., & Hutchison, K. E. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0706111104 · Cited by 683
Self-citationRoesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597
Hikida, T., Kimura, K., Wada, N., Funabiki, K., & Nakanishi, S. (2010) · Neuron · doi:10.1016/j.neuron.2010.05.011 · Cited by 592
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Midbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503
Frank, M. J., & O'Reilly, R. C. (2006) · Behavioral Neuroscience · doi:10.1037/0735-7044.120.3.497 · Cited by 472
Self-citationDisentangling pleasure from incentive salience and learning signals in brain reward circuitry
Smith, K. S., Berridge, K. C., & Aldridge, J. W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1101920108 · Cited by 464
Beyond the reward hypothesis: alternative functions of nucleus accumbens dopamine
Salamone, J., Correa, M., Mingote, S., & Weber, S. (2005) · Current Opinion in Pharmacology · doi:10.1016/j.coph.2004.09.004 · Cited by 458
Dopaminergic and Glutamatergic Regulation of Effort- and Delay-Based Decision Making
Floresco, S. B., Tse, M. T. L., & Ghods-Sharifi, S. (2008) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301565 · Cited by 435
Cognitive control over learning: Creating, clustering, and generalizing task-set structure.
Collins, A. G. E., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0030852 · Cited by 426
Self-citationA computational substrate for incentive salience
McClure, S. M., Daw, N. D., & Read Montague, P. (2003) · Trends in Neurosciences · doi:10.1016/s0166-2236(03)00177-2 · Cited by 422
Dopaminergic reward system: a short integrative review
Arias-Carrión, O., Stamelou, M., Murillo-Rodríguez, E., Menéndez-González, M., & Pöppel, E. (2010) · International Archives of Medicine · doi:10.1186/1755-7682-3-24 · Cited by 416
Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding
Huys, Q. J. M., Cools, R., Gölzer, M., Friedel, E., Heinz, A., Dolan, R. J., & Dayan, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002028 · Cited by 416
A computational model of inhibitory control in frontal cortex and basal ganglia.
Wiecki, T. V., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0031542 · Cited by 404
Self-citationTransient stimulation of distinct subpopulations of striatal neurons mimics changes in action value
Tai, L.-H., Lee, A. M., Benavidez, N., Bonci, A., & Wilbrecht, L. (2012) · Nature Neuroscience · doi:10.1038/nn.3188 · Cited by 399
Schönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2496-07.2007 · Cited by 390
Dopaminergic mechanisms of individual differences in human effort-based decision-making
Treadway, M. T., Buckholtz, J. W., Cowan, R. L., Woodward, N. D., Li, R., Ansari, M. S., Baldwin, R. M., Schwartzman, A. N., Kessler, R. M., & Zald, D. H. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.6459-11.2012 · Cited by 385
Cools, R., Frank, M. J., Gibbs, S. E., Miyakawa, A., Jagust, W., & D'Esposito, M. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4467-08.2009 · Cited by 364
Self-citationBodi, N., Keri, S., Nagy, H., Moustafa, A., Myers, C. E., Daw, N., Dibo, G., Takats, A., Bereczki, D., & Gluck, M. A. (2009) · Brain · doi:10.1093/brain/awp094 · Cited by 363
Molecular effects of dopamine on striatal-projection pathways
Gerfen, C. R. (2000) · Trends in Neurosciences · doi:10.1016/s1471-1931(00)00019-7 · Cited by 341
A Neural Computational Model of Incentive Salience
Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., & Aldridge, J. W. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000437 · Cited by 299 · cited 1×
“These findings support the observations that motivational state at the time of choice can impact an animal's behavior to approach an action which had been preferentially associated with negative outcomes during learning (Zhang et al., 2009).”
ResultCorrelated Coding of Motivation and Outcome of Decision by Dopamine Neurons
Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-30-09913.2003 · Cited by 298
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 286
Jocham, G., Klein, T. A., & Ullsperger, M. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3904-10.2011 · Cited by 278
Aberman, J. E., & Salamone, J. D. (1999) · Neuroscience · doi:10.1016/s0306-4522(99)00004-4 · Cited by 263
Cousins, M. S., Atherton, A., Turner, L., & Salamone, J. D. (1996) · Behavioural Brain Research · doi:10.1016/0166-4328(95)00151-4 · Cited by 236
Frank, M. J., Santamaria, A., O'Reilly, R. C., & Willcutt, E. (2007) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301278 · Cited by 227
Instructional control of reinforcement learning: A behavioral and neurocomputational investigation
Doll, B. B., Jacobs, W. J., Sanfey, A. G., & Frank, M. J. (2009) · Brain Research · doi:10.1016/j.brainres.2009.07.007 · Cited by 215
Self-citationRatcliff, R., & Frank, M. J. (2012) · Neural Computation · doi:10.1162/neco_a_00270 · Cited by 207
Self-citationA dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism
Moustafa, A. A., Sherman, S. J., & Frank, M. J. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2008.07.011 · Cited by 192
Self-citationStatistics of Midbrain Dopamine Neuron Spike Trains in the Awake Primate
Bayer, H. M., Lau, B., & Glimcher, P. W. (2007) · Journal of Neurophysiology · doi:10.1152/jn.01140.2006 · Cited by 187
Dopaminergic Genes Predict Individual Differences in Susceptibility to Confirmation Bias
Doll, B. B., Hutchison, K. E., & Frank, M. J. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.6486-10.2011 · Cited by 184
Self-citationTonic Dopamine Modulates Exploitation of Reward Learning
Beeler, J. A., Daw, N., Frazier, C. R. M., & Zhuang, X. (2010) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2010.00170 · Cited by 179
Cousins, M. S., & Salamone, J. D. (1994) · Pharmacology Biochemistry and Behavior · doi:10.1016/0091-3057(94)90460-x · Cited by 175
Temporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli
Nomoto, K., Schultz, W., Watanabe, T., & Sakagami, M. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4828-09.2010 · Cited by 158
Dopamine and performance in a reinforcement learning task: evidence from Parkinson’s disease
Shiner, T., Seymour, B., Wunderlich, K., Hill, C. S., Bhatia, K. P., Dayan, P., & Dolan, R. J. (2012) · Brain · doi:10.1093/brain/aws083 · Cited by 158
Wassum, K. M., Ostlund, S. B., Balleine, B. W., & Maidment, N. T. (2011) · Learning & Memory · doi:10.1101/lm.2229311 · Cited by 145
Mingote, S., Font, L., Farrar, A. M., Vontell, R., Worden, L. T., Stopper, C. M., Port, R. G., Sink, K. S., Bunce, J. G., Chrobak, J. J., et al. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1525-08.2008 · Cited by 141
Farrar, A. M., Segovia, K. N., Randall, P. A., Nunes, E. J., Collins, L. E., Stopper, C. M., Port, R. G., Hockemeyer, J., Müller, C. E., Correa, M., et al. (2010) · Neuroscience · doi:10.1016/j.neuroscience.2009.12.056 · Cited by 126
Farrar, A. M., Font, L., Pereira, M., Mingote, S., Bunce, J. G., Chrobak, J. J., & Salamone, J. D. (2008) · Neuroscience · doi:10.1016/j.neuroscience.2007.12.034 · Cited by 109
Neural mechanisms of acquired phasic dopamine responses in learning
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2009) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2009.11.019 · Cited by 107
Self-citationBrain Hemispheres Selectively Track the Expected Value of Contralateral Options
Palminteri, S., Boraud, T., Lafargue, G., Dubois, B., & Pessiglione, M. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1500-09.2009 · Cited by 99
A Role for Dopamine-Mediated Learning in the Pathophysiology and Treatment of Parkinson’s Disease
Beeler, J. A., Frank, M. J., McDaid, J., Alexander, E. R., Turkson, S., Sarria, M. S. B., McGehee, D. S., & Zhuang, X. (2012) · Cell Reports · doi:10.1016/j.celrep.2012.11.014 · Cited by 95
Self-citationNunes, E. J., Randall, P. A., Santerre, J. L., Given, A. M., Sager, T. N., Correa, M., & Salamone, J. D. (2010) · Neuroscience · doi:10.1016/j.neuroscience.2010.05.068 · Cited by 93
Smittenaar, P., Chase, H. W., Aarts, E., Nusselein, B., Bloem, B. R., & Cools, R. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08043.x · Cited by 75
Instrumental vigour in punishment and reward
Dayan, P. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08026.x · Cited by 73
Ott, D. V. M., Ullsperger, M., Jocham, G., Neumann, J., & Klein, T. A. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.04.038 · Cited by 52
Wiecki, T. V., Riedinger, K., von Ameln-Mayerhofer, A., Schmidt, W., & Frank, M. J. (2009) · Psychopharmacology · doi:10.1007/s00213-008-1457-4 · Cited by 47
Self-citationContext-dependent catalepsy intensification is due to classical conditioning and sensitization
Amtage, J., & Schmidt, W. (2003) · Behavioural Pharmacology · doi:10.1097/00008877-200311000-00009 · Cited by 46
Klein, A., & Schmidt, W. (2003) · Behavioural Pharmacology · doi:10.1097/00008877-200302000-00005 · Cited by 42
Cited by
Berridge, K. C., & Kringelbach, M. L. (2015) · Neuron · doi:10.1016/j.neuron.2015.02.018 · Cited by 1,730
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
Self-citationTen simple rules for the computational modeling of behavioral data
Wilson, R. C., & Collins, A. G. (2019) · eLife · doi:10.7554/elife.49547 · Cited by 710
Self-citationStriatal circuits for reward learning and decision-making
Cox, J., & Witten, I. B. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0189-2 · Cited by 568
Motor symptoms in Parkinson’s disease: A unified framework
Moustafa, A. A., Chakravarthy, S., Phillips, J. R., Gupta, A., Keri, S., Polner, B., Frank, M. J., & Jahanshahi, M. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.07.010 · Cited by 481
Self-citation
Metadata
- Authors
- Collins, A. G. E., & Frank, M. J.
- Venue
- Psychological Review
- Published
- 2014
- Volume
- 121
- Issue
- 3
- Pages
- 337-366
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- OpenAlex
- W1982670892
- Cited by
- 403
- References
- 68
- FWCI
- 9.93
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.