Subcortical Substrates of Explore-Exploit Decisions in Primates
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Summary The explore-exploit dilemma refers to the challenge of deciding when to forego immediate rewards and explore new opportunities that could lead to greater rewards in the future. While motivational neural circuits facilitate reinforcement learning based on past choices and outcomes, it is unclear if they also support computations relevant for deciding when to explore. We recorded neural activity in the amygdala and ventral striatum of rhesus macaques, as they solved a task that required them to balance novelty driven exploration with exploitation of what they had already learned. Using a partially-bserved Markov decision process model to quantify explore-exploit trade-offs, we identified that the ventral striatum and amygdala differ in how they represent the immediate value of exploitative choices and the future value of exploratory choices. These findings show that subcortical motivational circuits are important in guiding exploratory decisions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Aston-Jones, G., & Cohen, J. D. (2005) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.28.061604.135709 · Cited by 4,472
The reward circuit: linking primate anatomy and human imaging
Haber, S. N., & Knutson, B. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.129 · Cited by 3,718
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 1×
“There has been considerable focus on the striatum and dopamine system in RL (Costa et al., 2015; O'Doherty et al., 2004; Parker et al., 2016; Schultz, 2015).”
BackgroundPreliminary analysis of functions of the temporal lobes in monkeys
KLÜVER, H. (1939) · Archives of Neurology And Psychiatry · doi:10.1001/archneurpsyc.1939.02270240017001 · Cited by 1,696 · cited 1×
“, 2008) and monkeys with amygdala lesions more readily approach novel stimuli (Burns et al., 1996; Kluver and Bucy, 1939; Mason et al., 2006).”
BackgroundThe importance of mixed selectivity in complex cognitive tasks
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J., Daw, N. D., Miller, E. K., & Fusi, S. (2013) · Nature · doi:10.1038/nature12160 · Cited by 1,639 · cited 1×
“, 2018), similar to coding schemes in cortical areas (Rigotti et al., 2013).”
BackgroundOlejnik, S., & Algina, J. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.4.434 · Cited by 1,622
Bandit processes and dynamic allocation indices
Gittins, J. C. (1979) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1979.tb01068.x · Cited by 1,487 · cited 1×
“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript (RL) and while theoretical solutions exist (Averbeck, 2015; Gittins, 1979; Wilson et al., 2014a), their neural implementation has received limited attention (Cohen et al., 2007; Daw et al., 2006; Frank et al., 2009).”
BackgroundNeural systems responding to degrees of uncertainty in human decision-making
Hsu, M., Bhatt, M., Adolphs, R., Tranel, D., & Camerer, C. F. (2005) · Science · doi:10.1126/science.1115327 · Cited by 1,411
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030
The Psychology and Neuroscience of Curiosity
Kidd, C., & Hayden, B. Y. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.010 · Cited by 1,026
Natural selective attention: Orienting and emotion
Bradley, M. M. (2009) · Psychophysiology · doi:10.1111/j.1469-8986.2008.00702.x · Cited by 976
The Nucleus Accumbens: An Interface Between Cognition, Emotion, and Action
Floresco, S. B. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010213-115159 · Cited by 938
The primate amygdala represents the positive and negative value of visual stimuli during learning
Paton, J. J., Belova, M. A., Morrison, S. E., & Salzman, C. D. (2006) · Nature · doi:10.1038/nature04490 · Cited by 917 · cited 1×
“Although it is established that the amygdala (Averbeck and Costa, 2017; Belova et al., 2008; Morrison and Salzman, 2010; Paton et al., 2006) and ventral striatum (Cai et al.”
BackgroundOrbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885
Excitatory transmission from the amygdala to nucleus accumbens facilitates reward seeking
Stuber, G. D., Sparta, D. R., Stamatakis, A. M., van Leeuwen, W. A., Hardjoprajitno, J. E., Cho, S., Tye, K. M., Kempadoo, K. A., Zhang, F., Deisseroth, K., et al. (2011) · Nature · doi:10.1038/nature10194 · Cited by 878
A circuit mechanism for differentiating positive and negative associations
Namburi, P., Beyeler, A., Yorozu, S., Calhoon, G. G., Halbert, S. A., Wichmann, R., Holden, S. S., Mertens, K. L., Anahtar, M., Felix-Ortiz, A. C., et al. (2015) · Nature · doi:10.1038/nature14366 · Cited by 614 · cited 1×
“Amygdala projections to the ventral striatum are known to invigorate reward seeking, support positive reinforcement, and affect dopamine release (Jones et al., 2010; Namburi et al., 2015; Stuber et al., 2011).”
BackgroundHumans use directed and random exploration to solve the explore–exploit dilemma.
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0038199 · Cited by 580
Midbrain Dopamine Neurons Signal Preference for Advance Information about Upcoming Rewards
Bromberg-Martin, E. S., & Hikosaka, O. (2009) · Neuron · doi:10.1016/j.neuron.2009.06.009 · Cited by 548
Processing of Temporal Unpredictability in Human and Animal Amygdala
Herry, C., Bach, D. R., Esposito, F., Di Salle, F., Perrig, W. J., Scheffler, K., Lüthi, A., & Seifritz, E. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5218-06.2007 · Cited by 401
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277 · cited 5×
“The task design was based on a multi-arm bandit task previously used in humans and monkeys to study explore-exploit trade-offs (Costa et al., 2014; Djamshidian et al., 2011; Wittmann et al., 2008).”
BackgroundMethodResultInfluentialDopamine modulates novelty seeking behavior during decision making.
Costa, V. D., Tran, V. L., Turchi, J., & Averbeck, B. B. (2014) · Behavioral Neuroscience · doi:10.1037/a0037128 · Cited by 263 · cited 5×
“Increasing extracellular dopamine levels heightens novelty-driven exploration (Costa et al., 2014) and dopamine neuron responses to novelty decrease with learning (Lak et al.”
BackgroundMethodResultInfluentialSelf-citationDistributed and Mixed Information in Monosynaptic Inputs to Dopamine Neurons
Tian, J., Huang, R., Cohen, J. Y., Osakada, F., Kobak, D., Machens, C. K., Callaway, E. M., Uchida, N., & Watabe-Uchida, M. (2016) · Neuron · doi:10.1016/j.neuron.2016.08.018 · Cited by 224
Moment-to-moment tracking of state value in the amygdala
Belova, M. A., Paton, J. J., & Salzman, C. D. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1400-08.2008 · Cited by 221
A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research
Addicott, M. A., Pearson, J. M., Sweitzer, M. M., Barack, D. L., & Platt, M. L. (2017) · Neuropsychopharmacology · doi:10.1038/npp.2017.108 · Cited by 219
Abstract context representations in primate amygdala and prefrontal cortex
Saez, A., Rigotti, M., Ostojic, S., Fusi, S., & Salzman, C. D. (2015) · Neuron · doi:10.1016/j.neuron.2015.07.024 · Cited by 183 · cited 3×
“While It is unlikely that the monkeys had a perfect knowledge of the task statistics, they were overtrained on the task which could have led to them developing sophisticated model-based strategies (Costa et al., 2015; Jang et al., 2015; Saez et al., 2015; Seo et al., 2014).”
BackgroundResultReversal learning and dopamine: a Bayesian perspective
Costa, V. D., Tran, V. L., Turchi, J., & Averbeck, B. B. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1989-14.2015 · Cited by 170
Self-citationMulti-dimensional coding by basolateral amygdala neurons
Kyriazi, P., Headley, D. B., & Pare, D. (2018) · Neuron · doi:10.1016/j.neuron.2018.07.036 · Cited by 130 · cited 1×
“This result fits with increasing evidence that value coding in both structures is multidimensional and heterogenous (Cai et al., 2011; Gore et al., 2015; Kyriazi et al., 2018), similar to coding schemes in cortical areas (Rigotti et al.”
Amygdala signaling during foraging in a hazardous environment
Amir, A., Lee, S.-C., Headley, D. B., Herzallah, M. M., & Pare, D. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.0407-15.2015 · Cited by 104 · cited 1×
“While the amygdala appears to regulate foraging in aversive environments (Amir et al., 2015; Choi and Kim, 2010) and the ventral striatum contributes to value-based decision making (Floresco, 2015; Haber and Knutson, 2010), these structures might support different computations during foraging tasks than those we have…”
BackgroundMason, W. A., Capitanio, J. P., Machado, C. J., Mendoza, S. P., & Amaral, D. G. (2006) · Emotion · doi:10.1037/1528-3542.6.1.73 · Cited by 96
Sethi, A., Voon, V., Critchley, H. D., Cercignani, M., & Harrison, N. A. (2018) · Brain · doi:10.1093/brain/awy048 · Cited by 36 · cited 1×
“, 2018) and establish how cortical and subcortical motivational circuits interact to bias exploration in clinically relevant directions (Addicott et al., 2017; Averbeck et al., 2013a; Morris et al., 2016; Sethi et al., 2018).”
Background
Cited by
Deep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269
Medial and orbital frontal cortex in decision-making and flexible behavior
Klein-Flügge, M. C., Bongioanni, A., & Rushworth, M. F. S. (2022) · Neuron · doi:10.1016/j.neuron.2022.05.022 · Cited by 244
The anterior cingulate cortex directs exploration of alternative strategies
Tervo, D. G. R., Kuleshova, E., Manakov, M., Proskurin, M., Karlsson, M., Lustig, A., Behnam, R., & Karpova, A. Y. (2021) · Neuron · doi:10.1016/j.neuron.2021.03.028 · Cited by 97
Trudel, N., Scholl, J., Klein-Flügge, M. C., Fouragnan, E., Tankelevitch, L., Wittmann, M. K., & Rushworth, M. F. S. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0929-3 · Cited by 74
Curiosity: primate neural circuits for novelty and information seeking
Monosov, I. E. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00784-9 · Cited by 44
“…context-dependent modulation of value by novelty afforded by this architecture can help to solve some forms of exploration– exploitation dilemmas 3,12,19,62,73 , particularly in contexts in which rewards are sparse, by allowing for flexible boosting (or bonusing) of RPEs by novelty to modulate…”
Metadata
- Authors
- Costa, V. D., Mitz, A. R., & Averbeck, B. B.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2019-08
- Volume
- 103
- Issue
- 3
- Pages
- 533-545.e5
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2952908749
- Cited by
- 133
- References
- 32
- FWCI
- 7.84
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.