Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Evaluation of both immediate and future outcomes of one's actions is a critical requirement for intelligent behavior. Using functional magnetic resonance imaging (fMRI), we investigated brain mechanisms for reward prediction at different time scales in a Markov decision task. When human subjects learned actions on the basis of immediate rewards, significant activity was seen in the lateral orbitofrontal cortex and the striatum. When subjects learned to act in order to obtain large future rewards while incurring small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects' performance data revealed graded maps of time scale within the insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior regions were involved in predicting future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In neurotypical individuals, the key brain loci that have been implicated in prediction are basal ganglia (68, 69), striatum (70, 71), anterior cingulate (72), and cerebellum (73, 74).”
From Autism as a disorder of prediction (2014) · cites this paper as [69]
Citations
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“, which has been successfully used for explaining reward-predictive activations of the midbrain dopaminergic system as well as those of the cortex and the striatu”
MethodEmotion, Decision Making and the Orbitofrontal Cortex
Bechara, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.295 · Cited by 3,589 · cited 1×
“Damage to the prefrontal cortex often impairs daily decision making, which requires assessment of future outcome”
BackgroundBarto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Basal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“The cortex and the basal ganglia both have parallel loop organization, with four major loops (limbic, cognitive, motor and oculomotor) and finer, topographic sub-loops within each major loo”
Background(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
The Orbitofrontal Cortex and Reward
Rolls, E. T. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.284 · Cited by 1,595
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001) · Neuron · doi:10.1016/s0896-6273(01)00303-8 · Cited by 1,404
Cited by
Items 101–200 of 798
Neuromodulation of reward‐based learning and decision making in human aging
Eppinger, B., Hämmerer, D., & Li, S. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06230.x · Cited by 234
The subthalamic nucleus exerts opposite control on cocaine and 'natural' rewards
Baunez, C., Dias, C., Cador, M., & Amalric, M. (2005) · Nature Neuroscience · doi:10.1038/nn1429 · Cited by 223
Haber, S. N., & Behrens, T. E. J. (2014) · Neuron · doi:10.1016/j.neuron.2014.08.031 · Cited by 230
“…responding in working memory tasks are often found also in the VS, in addition to those in the dorsal striatum, (Apicella et al., 1991; Cromwell and Schultz, 2003; Delgado et al., 2005; Hassani et al., 2001; Levy et al., 1997; Takikawa et al., 2002; Tanaka et al., 2004; Watanabe et al., 2003).”
BackgroundSensitivity to reward and punishment and the prefrontal cortex in major depression
Must, A., Szabó, Z., Bódi, N., Szász, A., Janka, Z., & Kéri, S. (2006) · Journal of Affective Disorders · doi:10.1016/j.jad.2005.12.005 · Cited by 220
Neural systems underlying approach and avoidance in anxiety disorders
Robin L., A., & Martin, P. P. (2010) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2010.12.4/raupperle · Cited by 227
“Other studies support the dorsal PFC’s role in risk-taking and delayed-discounting.(29,31,33,241,242) Human neuroimaging research has partially supported the distinctions between ventral and dorsal PFC by providing evidence that, while OFC regions are important for calculating value of choices, dmPFC regions are invol…”
Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making
Kalenscher, T., & Pennartz, C. M. A. (2008) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2007.11.004 · Cited by 222
“Similar results have been found by Tanaka et al. (2004).”
ResultWave-like dopamine dynamics as a mechanism for spatiotemporal credit assignment
Hamid, A. A., Frank, M. J., & Moore, C. I. (2021) · Cell · doi:10.1016/j.cell.2021.03.046 · Cited by 226
“The model captures regional reward credit assignment in functionally specialized cortico-basal ganglia (BG) loops, inspired by previous anatomical and functional reports (Aoki et al., 2019; Barbera et al., 2016; Haber, 2003; Hintiryan et al., 2016; Hunnicutt et al., 2016; Klaus et al., 2017; Lee et al., 2020; Mandelba…”
MethodSkirrow, C., McLoughlin, G., Kuntsi, J., & Asherson, P. (2009) · Expert Review of Neurotherapeutics · doi:10.1586/ern.09.2 · Cited by 217
Common and distinct neural correlates of personal and vicarious reward: A quantitative meta-analysis
Morelli, S. A., Sacchet, M. D., & Zaki, J. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.056 · Cited by 203
Frontal Networks for Learning and Executing Arbitrary Stimulus-Response Associations
Boettiger, C. A., & D'Esposito, M. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3697-04.2005 · Cited by 213
“…FAM NOV areas are putatively engaged by monitoring the outcome of actions [ACC (Van Veen et al., 2001; Hadland et al., 2003)] or predicting reward [insula (Tanaka et al., 2004)], one could speculate that subjects were evaluating their performance in the task during the less demanding (FAM) blocks.”
BackgroundResultEncoding of Marginal Utility across Time in the Human Brain
Pine, A., Seymour, B., Roiser, J. P., Bossaerts, P., Friston, K. J., Curran, H. V., & Dolan, R. J. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1126-09.2009 · Cited by 169
“In humans, activity has been shown to correlate with preferences for immediate options (McClure et al., 2004) and for discounted magnitude across both immediate and delayed options over both short (Tanaka et al., 2004) and long timescales (Kable and Glimcher, 2007).”
BackgroundMethodInfluentialDiversity and plasticity of microglial cells in psychiatric and neurological disorders
Nakagawa, Y., & Chiba, K. (2015) · Pharmacology & Therapeutics · doi:10.1016/j.pharmthera.2015.06.010 · Cited by 208
Ainslie, G. (2005) · Behavioral and Brain Sciences · doi:10.1017/s0140525x05000117 · Cited by 201
Remember the future II: Meta-analyses and functional overlap of working memory and delay discounting
Wesley, M. J., & Bickel, W. K. (2014) · Biological Psychiatry · doi:10.1016/j.biopsych.2013.08.008 · Cited by 205
Resolution of Uncertainty in Prefrontal Cortex
Yoshida, W., & Ishii, S. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.006 · Cited by 197
“Although several recent studies using probabilistic models have been successful in reproducing internal information processing (Smith et al., 2004; Tanaka et al., 2004), the regression functions have often been calculated based on the maximum likelihood or maximum a posteriori inference for hidden variables, and stati…”
BackgroundOverlapping neural systems represent cognitive effort and reward anticipation
Vassena, E., Silvetti, M., Boehler, C. N., Achten, E., Fias, W., & Verguts, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0091008 · Cited by 200
The basal ganglia and the cerebellum in human emotion
Pierce, J. E., & Péron, J. (2020) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsaa076 · Cited by 191
Fusi, S., Asaad, W. F., Miller, E. K., & Wang, X.-J. (2007) · Neuron · doi:10.1016/j.neuron.2007.03.017 · Cited by 199
“In particular , the basal ganglia appears to play a major role, as shown by behavioral (Murray et al., 2000), physiological (Pasupathy and Miller, 2005), and imaging (Tanaka et al., 2004; Boettiger et al., 2005) studies.”
BackgroundResultBehavioral and Neural Properties of Social Reinforcement Learning
Jones, R. M., Somerville, L. H., Li, J., Ruberry, E. J., Libby, V., Glover, G., Voss, H. U., Ballon, D. J., & Casey, B. J. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2972-11.2011 · Cited by 191
“Measuring differences in reaction times to cues to index learning differs from reinforcement studies that use modulated choice behavior as an indicator of learning (Tanaka et al., 2004; Daw and Doya, 2006; Schönberg et al., 2007).”
BackgroundResultThe neural substrates of probabilistic and intertemporal decision making
Weber, B. J., & Huettel, S. A. (2008) · Brain Research · doi:10.1016/j.brainres.2008.07.105 · Cited by 178
Tanaka, S. C., Schweighofer, N., Asahi, S., Shishida, K., Okamoto, Y., Yamawaki, S., & Doya, K. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001333 · Cited by 184
“In our previous brain imaging study [13], we demonstrated topographic maps of the time scales of reward prediction in the insular cortex and striatum in support of the first hypothesis.”
ResultSelf-citationAn Update on the Role of Serotonin and its Interplay with Dopamine for Reward
Fischer, A. G., & Ullsperger, M. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00484 · Cited by 176
“Tanaka et al. (2004) found increased activity overlapping with the DRN, when participants had to endure short-term losses to obtain long-term rewards, akin to effects observed in the dorsal 1www.globaldrugsurvey.com Frontiers in Human Neuroscience | www.frontiersin.org 5 October 2017 | Volume 11 |…”
MethodCerebral cortex : principles of operation
Rolls, E. T. (2016) · doi:10.82444/warw.33712699 · Cited by 175
Neural Correlates of Forward Planning in a Spatial Decision Task in Humans
Simon, D. A., & Daw, N. D. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4647-10.2011 · Cited by 177
“…(Pagnoni et al., 2002; McClure et al., 2003; Yacubian et al., 2006; Hampton and O’Doherty, 2007) rather than value (but see Delgado et al., 2000; Tanaka et al., 2004; Kable and Glimcher, 2007; Tom et al., 2007; on unit physiology: Arkadir et al., 2004; Samejima et al., 2005; Kim et al., 2009).”
BackgroundMethodCai, X., Kim, S., & Lee, D. (2011) · Neuron · doi:10.1016/j.neuron.2010.11.041 · Cited by 181
Hunger Does Not Motivate Reward in Women Remitted from Anorexia Nervosa
Wierenga, C. E., Bischoff-Grethe, A., Melrose, A. J., Irvine, Z., Torres, L., Bailer, U. F., Simmons, A., Fudge, J. L., McClure, S. M., Ely, A., et al. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.09.024 · Cited by 171
“The DLPFC subregion of the MFG and the VLPFC are often associated with cognitive control, including numeric computation, future planning, and inhibition (37), and the insula is involved in the perception of time (38) and codes the selection of options for immediate versus delayed gratification (39).”
Weber, M. J., Messing, S. B., Rao, H., Detre, J. A., & Thompson‐Schill, S. L. (2014) · Human Brain Mapping · doi:10.1002/hbm.22429 · Cited by 183
Andrews, M. M., Meda, S. A., Thomas, A. D., Potenza, M. N., Krystal, J. H., Worhunsky, P., Stevens, M. C., O'Malley, S., Book, G. A., Reynolds, B., et al. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.09.049 · Cited by 181
Rudebeck, P. H., Bannerman, D. M., & Rushworth, M. F. S. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.485 · Cited by 172
“…dissociation has also been reported during tasks in which participants had to choose between different options on the basis of the costs and benefits associated with each (McClure, Ericson, Laibson, Loewenstein, & Cohen, 2007; McClure, Laibson, Loewenstein, & Cohen, 2004; Tanaka et al., 2004).”
BackgroundCho, S. S., Ko, J. H., Pellecchia, G., Van Eimeren, T., Cilia, R., & Strafella, A. P. (2010) · Brain Stimulation · doi:10.1016/j.brs.2009.10.002 · Cited by 179
“, orbitofrontal cortex-ventral striatum versus DLPFC-dorsal striatum) in relation to time-related decision making.(49,50) There was no significant modulation of DD value after the iTBS.”
Kohno, M., Morales, A. M., Ghahremani, D. G., Hellemann, G., & London, E. D. (2014) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.399 · Cited by 175
Henderson, L. A., Gandevia, S. C., & Macefield, V. G. (2007) · Pain · doi:10.1016/j.pain.2006.08.013 · Cited by 177
“Both the insula and striatum are also involved in predicting future rewards (Berns et al., 1997; Tanaka et al., 2004) and, as a result, the posterior insula is in an ideal position to determine and select appropriate motor responses to noxious stimuli.”
BackgroundFouragnan, E., Retzler, C., & Philiastides, M. G. (2018) · Human Brain Mapping · doi:10.1002/hbm.24047 · Cited by 176
Boorman, E. D., Rushworth, M. F., & Behrens, T. E. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.3022-12.2013 · Cited by 164
“One alternative possibility is that vmPFC, mid-cingulate, and PCC encode the net subjective value of the decision at hand (Tanaka et al., 2004; Daw et al., 2006; Hampton et al., 2006; Kable and Glimcher, 2007; Tom et al., 2007).”
BackgroundFellows, L. K. (2006) · Brain · doi:10.1093/brain/awl017 · Cited by 162
“…burgeoning functional imaging and electrophysiology literatures examining the role of ventral and medial prefrontal cortex in various aspects of decision making (Critchley et al., 2001; Ernst et al., 2002; Gehring and Willoughby, 2002; Krawczyk, 2002; McClure et al., 2004a; Tanaka et al., 2004).”
BackgroundA model-based fMRI analysis with hierarchical Bayesian parameter estimation.
Ahn, W.-Y., Krawitz, A., Kim, W., Busemeyer, J. R., & Brown, J. W. (2011) · Journal of Neuroscience, Psychology, and Economics · doi:10.1037/a0020684 · Cited by 162
Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement
Noonan, M. P., Kolling, N., Walton, M. E., & Rushworth, M. F. S. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08023.x · Cited by 169
van Heeringen, C., Bijttebier, S., & Godfrin, K. (2011) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2010.08.007 · Cited by 167
Jimura, K., Chushak, M. S., & Braver, T. S. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.0919-12.2013 · Cited by 153
“…and extend previous research indicating a link between VS activation and behavioral impulsivity, both in terms of intertemporal decision making (Tanaka et al., 2004; Hariri et al., 2006; Kable and Glimcher, 2007; Ballard and Knutson, 2009; Pine et al., 2009) and other trait markers (Dalley et…”
BackgroundThe lateral prefrontal cortex and complex value-based learning and decision making
Dixon, M. L., & Christoff, K. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.04.011 · Cited by 179
Modulations of the experience of self and time
Wittmann, M. (2015) · Consciousness and Cognition · doi:10.1016/j.concog.2015.06.008 · Cited by 181
The hippocampus, time, and memory across scales.
Howard, M. W., & Eichenbaum, H. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0033621 · Cited by 153
“For instance, Tanaka et al. (2004) correlated blood oxygenated level dependent (BOLD) response with the amount of reward expected at various time points in the future as predicted by reinforcement learning models with different parameters controlling the rate of temporal discounting.”
The role of serotonin in the regulation of patience and impulsivity
Miyazaki, K., Miyazaki, K. W., & Doya, K. (2012) · Molecular Neurobiology · doi:10.1007/s12035-012-8232-6 · Cited by 162
Self-citationEffects of chronic stress on cognitive function – From neurobiology to intervention
Girotti, M., Bulin, S. E., & Carreno, F. R. (2024) · Neurobiology of Stress · doi:10.1016/j.ynstr.2024.100670 · Cited by 209
“…basolateral amygdala (BLA), hippocampus, insula, lateral prefrontal cortex (PFC), posterior cingulate cortex, parietal cortex, and medial and lateral orbitofrontal OFC in delay discounting impulsivity (Cardinal et al., 2001; Kable and Glimcher 2007; Tanaka et al., 2004; Winstanley et al., 2004b).”
Neural mechanism of intertemporal choice: From discounting future gains to future losses
Xu, L., Liang, Z.-Y., Wang, K., Li, S., & Jiang, T. (2009) · Brain Research · doi:10.1016/j.brainres.2008.12.061 · Cited by 161
“Previous research on discounting future gains (Boettiger et al., 2007; McClure et al., 2004a; McClure et al., 2007; Monterosso et al., 2007; Tanaka et al., 2004) also found that the fronto-parietal network is important for temporal discounting, which accords well with its key role in executive…”
InfluentialPersonality predicts activity in reward and emotional regions associated with humor
Mobbs, D., Hagan, C. C., Azim, E., Menon, V., & Reiss, A. L. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0408457102 · Cited by 159
Signals in Human Striatum Are Appropriate for Policy Update Rather than Value Prediction
Li, J., & Daw, N. D. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.6316-10.2011 · Cited by 152
“…of significant value prediction-related activity in our study seems to contrast with many other studies in which striatal BOLD activity was demonstrably modulated by value expectation, as for prediction errors (Berns et al., 2001; O’Doherty et al., 2003; Tanaka et al., 2004; Hare et al., 2008).”
ResultMakris, N., Gasic, G. P., Kennedy, D. N., Hodge, S. M., Kaiser, J. R., Lee, M. J., Kim, B. W., Blood, A. J., Evins, A. E., Seidman, L. J., et al. (2008) · Neuron · doi:10.1016/j.neuron.2008.08.011 · Cited by 159
“…et al., 1997; Kringelbach, 2005; McClure et al., 2004; Mesulam and Mufson, 1985; Rolls et al., 2007; Schoenbaum et al., 1998; Seo et al., 2007; Tanaka et al., 2004; see Appendix 2, Data Analyses in the Supplemental Data for full set of references) and attention (ATTN) (Fan et al., 2005;…”
BackgroundCunningham, W. A., Espinet, S. D., DeYoung, C. G., & Zelazo, P. D. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.04.044 · Cited by 163
“, 2003) and basal ganglia (Delgado et al., 2000; Tanaka et al., 2004) are involved more in the processing of positive affect than negative effect (see Wager et al.”
BackgroundNeural Dissociation of Delay and Uncertainty in Intertemporal Choice
Luhmann, C. C., Chun, M. M., Yi, D.-J., Lee, D., & Wang, X.-J. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.5058-08.2008 · Cited by 150
“…previous neuroeconomic work has emphasized the neural representation of value of de- layed rewards (Cardinal et al., 2001; McClure et al., 2004; Tanaka et al., 2004; Cardinal, 2006; Hariri et al., 2006; Kable and Glimcher, 2007; McClure et al., 2007; Roesch et al., 2007; Wittmann et al.,…”
BackgroundThe Behavioral and Neural Mechanisms Underlying the Tracking of Expertise
Boorman, E. D., O’Doherty, J. P., Adolphs, R., & Rangel, A. (2013) · Neuron · doi:10.1016/j.neuron.2013.10.024 · Cited by 150
“Another open question about expertise tracking concerns the nature of the learning mechanism.”
BackgroundComputational models of basal-ganglia pathway functions: focus on functional neuroanatomy
Schroll, H., & Hamker, F. H. (2013) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2013.00122 · Cited by 134
“Empirical evidence on BG involvement in reinforcement learning is extensive for both animals (e.g., Featherstone and McDonald, 2004; El Massioui et al., 2007; Antzoulatos and Miller, 2011) and humans (e.g., Frank et al., 2004; Tanaka et al., 2004; Schönberg et al., 2007; Moustafa et al., 2008a).”
BackgroundDistinct Roles of Three Frontal Cortical Areas in Reward-Guided Behavior
Noonan, M. P., Mars, R. B., & Rushworth, M. F. S. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.6456-10.2011 · Cited by 145
“…been implicated in many rewardguided learning tasks, and it is responsive when reward and error feedback are provided that will allow a subject to update their reward-related associations (O’Doherty et al., 2004; Tanaka et al., 2004; Tobler et al., 2007; Rangel et al., 2008; Murayama et al., 2010).”
BackgroundThe human amygdala and orbital prefrontal cortex in behavioural regulation
Dolan, R. J. (2007) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2007.2088 · Cited by 150
“Thus, these findings can potentially explain the feeding abnormalities observed in both the Kluver– Bucy syndrome (Bechara et al. 1994) and frontotemporal dementias (Terzian & Ore 1955).”
BackgroundReinforcement learning: Computational theory and biological mechanisms
Doya, K. (2007) · HFSP journal · doi:10.2976/1.2732246 · Cited by 150
“3 (Tanaka et al. , 2004).”
Self-citationCoding of Task Reward Value in the Dorsal Raphe Nucleus
Bromberg-Martin, E. S., Hikosaka, O., & Nakamura, K. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.0015-10.2010 · Cited by 147
“In addition to these functions, there is evidence that the dorsal raphe nucleus regulates motivated behavior (Rompre and Miliaressis, 1985; Diotte et al., 2001; Tanaka et al., 2004; Chamberlain et al., 2006; Liu and Ikemoto, 2007).”
BackgroundDifferential roles of caudate nucleus and putamen during instrumental learning
Brovelli, A., Nazarian, B., Meunier, M., & Boussaoud, D. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.05.059 · Cited by 142
“…acquisition and consolidation of instrumental actions (Deiber et al., 1997; Toni and Passingham, 1999; Toni et al., 2001a,b; Haruno et al., 2004; Tanaka et al., 2004; Boettiger and D'Esposito, 2005; Delgado et al., 2005; Law et al., 2005; Grol et al., 2006; Haruno and Kawato, 2006;…”
Harris, A. M., Hare, T. A., & Rangel, A. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.5816-12.2013 · Cited by 156
“…state-dependent interoceptive processing specific to dietary choice, recent work suggests this region encodes more complex concepts, such as objective inequality in the ultimatum game (Hsu et al., 2008; Wright et al., 2011) and reward prediction in intertemporal choice (Tanaka et al., 2004).”
BackgroundTan, H.-Y., Chen, Q., Goldberg, T. E., Mattay, V. S., Meyer-Lindenberg, A., Weinberger, D. R., & Callicott, J. H. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.4041-07.2007 · Cited by 144
van Heeringen, K., Bijttebier, S., Desmyter, S., Vervaet, M., & Baeken, C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00824 · Cited by 141
Functional role of the ventrolateral prefrontal cortex in decision making
Sakagami, M., & Pan, X. (2007) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2007.02.008 · Cited by 141
How hierarchical models improve point estimates of model parameters at the individual level
Katahira, K. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2016.03.007 · Cited by 127
Ito, M., & Doya, K. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.04.001 · Cited by 160
Self-citationDissociable networks for the expectancy and perception of emotional stimuli in the human brain
Bermpohl, F., Pascual‐Leone, Á., Amedi, A., Merabet, L. B., Fregni, F., Gaab, N., Alsop, D. C., Schlaug, G., & Northoff, G. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.09.040 · Cited by 141
“…(Phillips et al., 2003b), taste reward (O'Doherty et al., 2002), monetary reward (Breiter et al., 2001; Knutson et al., 2001; Kahn et al., 2002; Kirsch et al., 2003; Knutson et al., 2003; Tanaka et al., 2004), and pain (Reiman et al., 1989; Ploghaus et al., 2003; Singer et al., 2004).”
BackgroundMahon, K., Burdick, K. E., Wu, J., Ardekani, B. A., & Szeszko, P. R. (2012) · Bipolar Disorders · doi:10.1111/j.1399-5618.2012.00984.x · Cited by 131
“Previous research has demonstrated that the left OFC is involved in the representation of the value of a reinforcer (78), in the selection of a goal-directed response (79), and in the prediction of an immediate reward (80).”
MethodMultiple Representations of Belief States and Action Values in Corticobasal Ganglia Loops
SAMEJIMA, K., & DOYA, K. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.024 · Cited by 124
Self-citationReward processing by the dorsal raphe nucleus: 5-HT and beyond
Luo, M., Zhou, J., & Liu, Z. (2015) · Learning & Memory · doi:10.1101/lm.037317.114 · Cited by 115
“Tanaka et al. (2004) showed that the DRN is activated prior to a large future gain together with a small immediate loss in human subjects.”
BackgroundLearning-Induced Plasticity in Medial Prefrontal Cortex Predicts Preference Malleability
Garvert, M. M., Moutoussis, M., Kurth-Nelson, Z., Behrens, T. E. J., & Dolan, R. J. (2015) · Neuron · doi:10.1016/j.neuron.2014.12.033 · Cited by 125
“been found in the human striatum (Tanaka et al., 2004) and mPFC (Wang et al.”
BackgroundValentin, V. V., & O'Doherty, J. P. (2009) · Journal of Neurophysiology · doi:10.1152/jn.91195.2008 · Cited by 117
“51 52 3 Introduction 53 54 Accumulating evidence from neurophysiological recording studies in nonhuman primates 55 and rodents implicates the phasic responses of dopamine neurons in encoding 56 prediction errors for reward (Mirenowicz and Schultz 1994; Morris et al. 2006; Roesch et 57 al.”
BackgroundHyperbolic Discounting and Learning over Multiple HorizonsPreprint
Fedus, W., Gelada, C., Bengio, Y., Bellemare, M. G., & Larochelle, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06865 · Cited by 140
“…mirrors the work of Kurth-Nelson & Redish (2009); Redish & Kurth-Nelson (2010) which empirically demonstrates that modeling a finite set of µ Agents simultaneously can approximate hyperbolic discounting function which is consistent with fMRI studies (Tanaka et al., 2004; Schweighofer et al., 2008).”
ResultNeuroanatomy of Reward: A View from the Ventral Striatum
Haber, S. N. (2011) · Europe PMC (PubMed Central) · Cited by 172
Functional neuroimaging of intertemporal choice models: a review
Carter, R. M., Meyer, J. R., & Huettel, S. A. (2010) · Journal of Neuroscience Psychology and Economics · doi:10.1037/a0018046 · Cited by 124
“…to years, with the remainder using much shorter delays of seconds to minutes for rewards obtained during the experiment itself (McClure et al., 2007) and for symbolic rewards that are accumulated (Gregorios-Pippas, Tobler, & Schultz, 2009; Luhmann, Chun, Yi, Lee, & Wang, 2008; Tanaka et al., 2004).”
BackgroundResultInfluentialQin (2011) · doi:10.1016/j.neuroimage.2011.05.006 · Cited by 124
Bickel, W. K., & Yi, R. (2008) · Advances in Health Economics and Health Services Research · doi:10.1016/s0731-2199(08)20012-9 · Cited by 122
Campos, A., Port, J. D., & Acosta, A. J. (2022) · Brain Sciences · doi:10.3390/brainsci12040431 · Cited by 139
Ventral–striatal/nucleus–accumbens sensitivity to prediction errors during classification learning
Rodriguez, P. F., Aron, A. R., & Poldrack, R. A. (2006) · Human Brain Mapping · doi:10.1002/hbm.20186 · Cited by 122
“, 2003] or instrumental conditioning [O’Doherty et al., 2004; Tanaka et al., 2004], finding that activity in the ventral striatum correlates with prediction error.”
BackgroundDissociating hippocampal and striatal contributions to sequential prediction learning
Bornstein, A. M., & Daw, N. D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07920.x · Cited by 124
“However, simply seeking the correlates of these time-series in a GLM (Tanaka et al., 2004; Gläscher & Buchel, 2005) is statistically inefficient because the versions of our variables of interest calculated at different values of a are highly correlated (average correlation across participants between slow and fast lea…”
Adaptive effort investment in cognitive and physical tasks: A neurocomputational model
Verguts, T., Vassena, E., & Silvetti, M. (2015) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2015.00057 · Cited by 118
“Other options include choosing between model-based and model-free processing (Daw et al., 2005), tolerating punishment in the face of upcoming rewards (Tanaka et al., 2004), or increasing cortical learning rate (Behrens et al., 2007; Silvetti et al., 2013a).”
BackgroundSynaptic plasticity in the basal ganglia
Wickens, J. R. (2008) · Behavioural Brain Research · doi:10.1016/j.bbr.2008.10.030 · Cited by 121
Kunisato, Y., Okamoto, Y., Ueda, K., Onoda, K., Okada, G., Yoshimura, S., Suzuki, S.-I., Samejima, K., & Yamawaki, S. (2012) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2012.05.007 · Cited by 134
“…along with a reinforcement learning model to analyze the results (Behrens, Woolrich, Walton, & Rushworth, 2007; Daw, O’Doherty, Dayan, Seymour, & Dolan, 2006; Frank, Moustafa, Haughey, Curran, & Hutchison, 2007; O’Doherty et al., 2004; Samejima, Ueda, Doya, & Kimura, 2005; Tanaka et al., 2004).”
Self-citationToward an Understanding of Decision Making in Severe Mental Illness
Cáceda, R., Nemeroff, C. B., & Harvey, P. D. (2014) · The Journal of Neuropsychiatry and Clinical Neurosciences · doi:10.1176/appi.neuropsych.12110268 · Cited by 110
“tive function, paradigm of cold cognition, reasoning, and planning, is associated with the activity of the right DLPFC and posterior parietal cortex.(29) The role of the PFC is further illustrated by the overwhelming preference for”
BackgroundGao, X., Yu, H., Sáez, I., Blue, P. R., Zhu, L., Hsu, M., & Zhou, X. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1802523115 · Cited by 132
Cerebral Correlates of Salient Prediction Error for Different Rewards and Punishments
Metereau, E., & Dreher, J.-C. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs037 · Cited by 114
“…between conditioned stimuli and different types of rewards, such as juice (Berns et al. 2001; McClure et al. 2003; O’Doherty et al. 2004), odor (Gottfried et al. 2003), money (Tanaka et al. 2004; Abler et al. 2006; Pessiglione et al. 2006), and attractive faces (Bray and O’Doherty 2007).”
The role of the dorsal raphé nucleus in reward-seeking behavior
Nakamura, K. (2013) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2013.00060 · Cited by 99
“In reinforcement learning theory, the state value is discounted when the delivery of the reward is delayed, and Doya et al. suggested that 5-HT regulates this reward discounting rate (Doya, 2002; Tanaka et al., 2004).”
BackgroundBernard, J. A., Russell, C. E., Newberry, R. E., Goen, J. R. M., & Mittal, V. A. (2017) · NeuroImage Clinical · doi:10.1016/j.nicl.2017.01.034 · Cited by 118
Cho, S. S., Koshimori, Y., Aminian, K., Obeso, I., Rusjan, P., Lang, A. E., Daskalakis, Z. J., Houle, S., & Strafella, A. P. (2015) · Neuropsychopharmacology · doi:10.1038/npp.2014.211 · Cited by 112
“Functional imaging studies using DD tasks are beginning to examine which brain areas are associated with impulsivity and self-control (McClure et al, 2004; Tanaka et al, 2004; Kable and Glimcher, 2007; Ballard and Knutson, 2009).”
BackgroundReward signals in the cerebellum: Origins, targets, and functional implications
Kostadinov, D., & Häusser, M. (2022) · Neuron · doi:10.1016/j.neuron.2022.02.015 · Cited by 119
Striatal-insula circuits in cocaine addiction: implications for impulsivity and relapse risk
McHugh, M. J., Demers, C. H., Braud, J., Briggs, R., Adinoff, B., & Stein, E. A. (2013) · The American Journal of Drug and Alcohol Abuse · doi:10.3109/00952990.2013.847446 · Cited by 110
“observed as a function of choosing the delayed over the immediate reward (67,68) as well as performance that relies upon the prediction of delayed rewards (69).”
BackgroundReward and Social Valuation Deficits following Ventromedial Prefrontal Damage
Moretti, L., Dragone, D., & di Pellegrino, G. (2009) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21011 · Cited by 107
Kim, B. K., & Zauberman, G. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0028954 · Cited by 106
Computational motor control in humans and robots
Schaal, S., & Schweighofer, N. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.10.009 · Cited by 115
The Cognitive Neuroscience of Motivation and Learning
Daw, N. D., & Shohamy, D. (2008) · Social Cognition · doi:10.1521/soco.2008.26.5.593 · Cited by 102
Delay discounting and intake of ready-to-eat and away-from-home foods in overweight and obese women
Appelhans, B. M., Waring, M. E., Schneider, K. L., Pagoto, S. L., DeBiasse, M. A., Whited, M. C., & Lynch, E. B. (2012) · Appetite · doi:10.1016/j.appet.2012.07.009 · Cited by 102
“The appetitive system largely consists of the mesolimbic dopamine pathway (Hariri et al., 2006; Tanaka et al., 2004), a neural circuit that mediates the hedonic motivation to engage in a number of appetitive behaviors including sex, social interaction, eating, gambling, and illicit drug use (Alcaro, Huber, & Panksepp,…”
BackgroundInfluentialSobel, L. J., Bansal, R., Maia, T. V., Sanchez, J., Mazzone, L., Durkin, K., Liu, J., Hao, X., Ivanov, I., Miller, A., et al. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091259 · Cited by 105
Dopamine transients follow a striatal gradient of reward time horizons
Mohebi, A., Wei, W., Pelattini, L., Kim, K., & Berke, J. D. (2024) · Nature Neuroscience · doi:10.1038/s41593-023-01566-3 · Cited by 114
“Although some imaging studies have suggested that VS circuits discount especially rapidly 33 , our results are instead consistent with an extensive literature demonstrating a critical role for VS in avoiding impulsive behavior 70 , by promoting work to obtain delayed rewards 71–73 .”
BackgroundResultShort-Term Temporal Discounting of Reward Value in Human Ventral Striatum
Gregorios-Pippas, L., Tobler, P. N., & Schultz, W. (2009) · Journal of Neurophysiology · doi:10.1152/jn.90730.2008 · Cited by 101
“All human reward discounting studies identified BOLD response discounting in the ventral striatum (Hariri et al. 2006; Kable and Glimcher 2007; McClure et al. 2004, 2007; Tanaka et al. 2004; Wittmann et al. 2007).”
BackgroundMethodInfluentialAn insula hierarchical network architecture for active interoceptive inference
Fermin, A. S. R., Friston, K., & Yamawaki, S. (2022) · Royal Society Open Science · doi:10.1098/rsos.220226 · Cited by 101
Self-citationNakao, T., Ohira, H., & Northoff, G. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00031 · Cited by 98
“Prediction error is related to the ventral striatum and the dorsal anterior cingulate cortex (dACC; Tanaka et al., 2004; Daw et al., 2006; Kim et al., 2006; Cohen, 2007; O’Doherty, 2007; Tom et al., 2007; Rolls et al., 2008; Glascher et al., 2009; Wunderlich et al., 2009, 2011).”
BackgroundJones, K. T., Gözenman, F., & Berryhill, M. E. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.11.012 · Cited by 98
“Lastly, the basal ganglia are associated with reward processing (Frank et al., 2001; Sesack and Grace, 2010; Tanaka et al., 2004; Vitay and Hamker, 2014), which further accentuates the possibility of anodal tDCS impacting WM tasks that have monetary rewards.”
Multiple memory systems as substrates for multiple decision systems
Doll, B. B., Shohamy, D., & Daw, N. D. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.04.014 · Cited by 95
“…and ventromedial prefrontal areas), it has been implicated in several studies with elaborated task preparations (Lee, Ghim, Kim, Lee, & Jung, 2012; Tanaka et al., 2004; Wimmer, Daw, & Shohamy, 2012; Wimmer & Shohamy, 2012), including those formally examining model-based decisions (Bornstein &…”
BackgroundResult
Metadata
- Authors
- Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Japan Science and Technology Agency (JP)
- Hiroshima University (JP)
- Corresponding author
- Doya, Kenji
- Venue
- Nature Neuroscience
- Published
- 2004-07-04
- Volume
- 7
- Issue
- 8
- Pages
- 887-893
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2032610016
- Cited by
- 779
- References
- 54
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 9 · 2022: 29 · 2021: 28 · 2020: 32 · 2019: 25 · 2018: 32 · 2017: 37 · 2016: 40 · 2015: 45 · 2014: 50 · 2013: 42 · 2012: 47
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- reward prediction, time scales, cortico-basal ganglia loop, prediction error, insula, striatum, orbitofrontal cortex, dorsolateral prefrontal cortex, dorsal raphe nucleus, cerebellum, computational modeling, reinforcement learning
- MeSH terms
- Adult, Brain, Brain Mapping, Discrimination Learning, Female, Humans, Magnetic Resonance Imaging, Male, Markov Chains, Middle Aged, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.