Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Evaluation of both immediate and future outcomes of one's actions is a critical requirement for intelligent behavior. Using functional magnetic resonance imaging (fMRI), we investigated brain mechanisms for reward prediction at different time scales in a Markov decision task. When human subjects learned actions on the basis of immediate rewards, significant activity was seen in the lateral orbitofrontal cortex and the striatum. When subjects learned to act in order to obtain large future rewards while incurring small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects' performance data revealed graded maps of time scale within the insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior regions were involved in predicting future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In neurotypical individuals, the key brain loci that have been implicated in prediction are basal ganglia (68, 69), striatum (70, 71), anterior cingulate (72), and cerebellum (73, 74).”
From Autism as a disorder of prediction (2014) · cites this paper as [69]
Citations
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“, which has been successfully used for explaining reward-predictive activations of the midbrain dopaminergic system as well as those of the cortex and the striatu”
MethodEmotion, Decision Making and the Orbitofrontal Cortex
Bechara, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.295 · Cited by 3,589 · cited 1×
“Damage to the prefrontal cortex often impairs daily decision making, which requires assessment of future outcome”
BackgroundBarto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Basal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“The cortex and the basal ganglia both have parallel loop organization, with four major loops (limbic, cognitive, motor and oculomotor) and finer, topographic sub-loops within each major loo”
Background(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
The Orbitofrontal Cortex and Reward
Rolls, E. T. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.284 · Cited by 1,595
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001) · Neuron · doi:10.1016/s0896-6273(01)00303-8 · Cited by 1,404
Cited by
Items 301–400 of 798
Serotonin Affects Association of Aversive Outcomes to Past Actions
Tanaka, S. C., Shishida, K., Schweighofer, N., Okamoto, Y., Yamawaki, S., & Doya, K. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2799-09.2009 · Cited by 64
“S5, available at www.jneurosci.org as supplemental material).”
ResultSelf-citationAging and the neuroeconomics of decision making: A review
Brown, S. B. R. E., & Ridderinkhof, K. R. (2009) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.9.4.365 · Cited by 70
“Serotonergic neurons project into the basal ganglia, controlling phasic dopamine release into the striatum, and thereby the time scale of reward prediction, modulating the adequate computation of reward prediction errors (Schweighofer et al., 2007; Tanaka et al., 2004).”
BackgroundEarly-life adversity and adolescent depression: mechanisms involving the ventral striatum
Goff, B., & Tottenham, N. (2015) · CNS Spectrums · doi:10.1017/s1092852914000674 · Cited by 61
The role of the dorsal striatum in choice impulsivity
Kim, B. S., & Im, H. (2018) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13961 · Cited by 65
Independent Neural Computation of Value from Other People's Confidence
Campbell-Meiklejohn, D. K., Simonsen, A., Frith, C., & Daw, N. D. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.4490-15.2016 · Cited by 70
“However, rather than covarying with the tendency to choose options on the basis of a dimension such as color, BOLD activity is widely reported to vary along the dimension of the chosen versus unchosen option, with larger responses in medial PFC when the chosen (or about to be chosen) option is more likely to be correc…”
BackgroundInfluentialMétéreau, É., & Dreher, J. (2014) · Cortex · doi:10.1016/j.cortex.2014.08.012 · Cited by 64
“…O'Doherty, & Rangel, 2011; Lim, O'Doherty, & Rangel, 2011; Lin, Adolphs, & Rangel, 2012; Litt, Plassmann, Shiv, & Rangel, 2011; Noonan et al., 2011; Plassmann et al., 2010, 2007; Prevost et al., 2010; Sescousse et al., 2010, 2014; Tanaka et al., 2004; Wunderlich, Rangel, & O'Doherty, 2010).”
Stimulus-preceding negativity is modulated by action-outcome contingency
Masaki, H., Yamazaki, K., & Hackley, S. A. (2010) · Neuroreport · doi:10.1097/wnr.0b013e3283360bc3 · Cited by 61
Brain networks of perceptual decision-making: an fMRI ALE meta-analysis
Keuken, M. C., Müller-Axt, C., Langner, R., Eickhoff, S. B., Forstmann, B. U., & Neumann, J. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00445 · Cited by 64
“In addition, there is a large body of literature on the involvement of the BG in reward-based decisionmaking (Kawagoe et al., 1998; Tanaka et al., 2004; Liu et al., 2011; Mulder et al., 2013).”
BackgroundInsulin, Central Dopamine D2 Receptors, and Monetary Reward Discounting in Obesity
Eisenstein, S. A., Gredysa, D. M., Antenor–Dorsey, J. A., Green, L., Arbeláez, A. M., Koller, J. M., Black, K. J., Perlmutter, J. S., Moerlein, S. M., & Hershey, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0133621 · Cited by 54
“Although several human neuroimaging studies link discounting behavior, including subjective value of delayed monetary reward [73–74] and coding of reward magnitude [75] to ventral striatal reactivity, the dorsal striatum is thought to be involved in future reward prediction, with a particular role in encoding temporal…”
BackgroundForgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation
Kato, A., & Morita, K. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005145 · Cited by 62
From ‘Understanding the Brain by Creating the Brain’ towards manipulative neuroscience
Kawato, M. (2008) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2008.2272 · Cited by 65
Skottnik, L., Sorger, B., Kamp, T., Linden, D. E. J., & Goebel, R. (2019) · Brain and Behavior · doi:10.1002/brb3.1240 · Cited by 61
Jiang, H., Zhu, R., Tian, S., Wang, H., Chen, Z., Wang, X., Shao, J., Qin, J., Shi, J., Liu, H., et al. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-020-0753-5 · Cited by 60
“Frontal alterations are thought to contribute to dysfunction in executive control of mood [38] and emotional pain processing [39], probably related to suicide [40, 41].”
BackgroundHierarchical Brain Networks Active in Approach and Avoidance Goal Pursuit
Spielberg, J. M., Heller, W., & Miller, G. A. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00284 · Cited by 64
Morita, K., Morishima, M., Sakai, K., & Kawaguchi, Y. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.4614-12.2013 · Cited by 65
Desrochers, T. M., Amemori, K.-I., & Graybiel, A. M. (2015) · Neuron · doi:10.1016/j.neuron.2015.07.019 · Cited by 54
“…the specific coding of the difference between the actual and expected reward, reward prediction error (RPE), has been found in the striatum of humans (Daw et al., 2006; O'Doherty et al., 2004; Tanaka et al., 2004), monkeys (Asaad and Eskandar, 2011), and rodents (Stalnaker et al., 2012).”
InfluentialNeural Segregation of Objective and Contextual Aspects of Fairness
Wright, N. D., Symmonds, M., Fleming, S. M., & Dolan, R. J. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.3138-10.2011 · Cited by 61
“Capacity of posterior insula for high-level computation is suggested by involvement in other high-level tasks, albeit different to fairness, for example, intertemporal choice (Tanaka et al., 2004; Wittmann et al., 2007) and language perception (Jones et al., 2010).”
BackgroundBrain and autonomic association accompanying stochastic decision-making
Ohira, H., Ichikawa, N., Nomura, M., Isowa, T., Kimura, K., Kanayama, N., Fukuyama, S., Shinoda, J., & Yamada, J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.07.060 · Cited by 59
“…activation with the putamen (Postuma and Dagher, 2006), has shown involvement in decision-making, especially when delivery of reward was delayed (McClure et al., 2004, 2007) and when the task required prediction of a future large reward and bearing immediate small loss (Tanaka et al., 2004).”
van den Bos, R., Davies, W., Dellu-Hagedorn, F., Goudriaan, A. E., Granon, S., Homberg, J. R., Rivalan, M., Swendsen, J. D., & Adriani, W. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.07.005 · Cited by 58
Wimmer, G. E., Li, J. K., Gorgolewski, K. J., & Poldrack, R. A. (2018) · Journal of Neuroscience · doi:10.1523/jneurosci.0075-18.2018 · Cited by 62
Autism: Oxytocin, serotonin, and social reward
Dölen, G. (2015) · Social Neuroscience · doi:10.1080/17470919.2015.1087875 · Cited by 52
“…has been suggested that DA and serotonin may encode reward on different timescales, such that DA encodes acutely reinforcing stimuli, while serotonin is required for delayed gratification rewards (Doya, 2002; Fonseca, Murakami, & Mainen, 2015; Miyazaki, Miyazaki, & Doya, 2012; Tanaka et al., 2004).”
BackgroundTemporal horizons in decision making.
Wittmann, M., & Paulus, M. P. (2009) · Journal of Neuroscience Psychology and Economics · doi:10.1037/a0015460 · Cited by 63
Serotonergic modulation of reward and punishment: evidence from pharmacological fMRI studies
Macoveanu, J. (2014) · Brain Research · doi:10.1016/j.brainres.2014.02.003 · Cited by 55
Local pattern classification differentiates processes of economic valuation
Clithero, J. A., Carter, R. M., & Huettel, S. A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.12.074 · Cited by 54
“There is a comparatively smaller collection of neuroimaging studies of intertemporal choice (Hariri et al., 2006; Kable and Glimcher, 2007; McClure et al., 2007; McClure et al., 2004; Tanaka et al., 2004; Weber and Huettel, 2008).”
BackgroundVoon, V. (2014) · Current Addiction Reports · doi:10.1007/s40429-014-0036-5 · Cited by 59
Impulsivity and Parkinson's disease: More than just disinhibition
Antonelli, F., Ray, N. J., & Strafella, A. P. (2011) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2011.06.006 · Cited by 58
“In addition, a study by Tanaka and colleagues [34] suggests that the striatum and the insula might be implicated in the evaluation of reward outcomes as a function of delay.”
ResultImaging studies of the HIV-infected brain
Chang, L., & Shukla, D. K. (2018) · Handbook of clinical neurology · doi:10.1016/b978-0-444-63849-6.00018-9 · Cited by 59
The Actor-Critic Learning Is Behind the Matching Law: Matching Versus Optimal Behaviors
Sakai, Y., & Fukai, T. (2007) · Neural Computation · doi:10.1162/neco.2008.20.1.227 · Cited by 61
“The motor-related frontal cortices and the striatum, an input nucleus of the basal ganglia, are considered to be essential for motor selection, thus operating as the actor (Houk, Davis, & Beiser, 1994; Doya, 2000; Dayan & Balleine, 2002; Tanaka et al., 2004; Schultz, 2004; Haruno et al., 2004).”
BackgroundZimmermann, K., Yao, S., Heinz, M., Zhou, F., Dau, W., Banger, M., Weber, B., Hurlemann, R., & Becker, B. (2017) · Psychopharmacology · doi:10.1007/s00213-017-4803-6 · Cited by 58
van Hasselt, F. N., de Visser, L., Tieskens, J. M., Cornelisse, S., Baars, A. M., Lavrijsen, M., Krugers, H. J., van den Bos, R., & Joëls, M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0037820 · Cited by 49
“In this scenario, the mPFC contributes to cognitive control over emotional influences on behavior, allowing the subject to maintain a long-term perspective and withhold responding to immediate rewards or losses [76,77].”
BackgroundReward-Predictive Neural Activities in Striatal Striosome Compartments
Yoshizawa, T., Ito, M., & Doya, K. (2018) · eneuro · doi:10.1523/eneuro.0367-17.2018 · Cited by 55
“Many lines of research, including functional brain imaging (O’Doherty et al., 2004; Tanaka et al., 2004) and neural recording (Samejima et al., 2005; Ito and Doya, 2009, 2015; Kim et al., 2009) have demonstrate that the striatum plays a critical role in decisionmaking and reinforcement learning.”
BackgroundSelf-citationYeh, Y.-W., Ho, P.-S., Chen, C.-Y., Kuo, S.-C., Liang, C.-S., Ma, K.-H., Shiue, C.-Y., Huang, W.-S., Cheng, C.-Y., Wang, T.-Y., et al. (2015) · International Journal of Neuropsychopharmacology · doi:10.1093/ijnp/pyu065 · Cited by 48
“In addition, the basal ganglia are associated with reward prediction, which is involved in decision-making (Tanaka et al., 2004).”
BackgroundMonosov, I. E., & Rushworth, M. F. S. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01079-2 · Cited by 55
Neural Evidence for Individual and Cultural Variability in the Social Comparison Effect
Kang, P., Lee, Y., Choi, I., & Kim, H. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.5084-12.2013 · Cited by 57
“A large body of literature supports the primary role of the vmPFC in encoding subjective values critical for guiding decisions in various situations (Tanaka et al., 2004; Kim et al., 2006; Kable and Glimcher, 2007; Kim et al., 2007; Chib et al., 2009) and also in encoding relative rather than…”
BackgroundPrediction of Immediate and Future Rewards Differentially Recruits Cortico-Basal Ganglia Loops
Tanaka, S., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2016) · Springer eBooks · doi:10.1007/978-4-431-55402-8_22 · Cited by 59
Self-citationIkeda, T., & Hikosaka, O. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00975.2007 · Cited by 58
“Recent studies have suggested that different brain areas are involved in aiming at immediate and delayed rewards, respectively (Hariri et al. 2006; Kobayashi et al. 2007; McClure et al. 2004; Roesch et al. 2006; Tanaka et al. 2004; Winstanley et al. 2004).”
BackgroundSimultaneous projections from prefrontal cortex to dopaminergic and serotonergic nuclei
Vázquez-Borsetti, P., Celada, P., Cortés, R., & Artigas, F. (2011) · The International Journal of Neuropsychopharmacology · doi:10.1017/s1461145710000349 · Cited by 46
“Dopamine neurons increase their activity to signal rewards or reward-associated stimuli (Schultz, 2004, 2007) and increases in DR activity have also been reported in response to future rewards (Tanaka et al. 2004).”
BackgroundLloyd, K., & Dayan, P. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004622 · Cited by 50
Gallucci, A., Riva, P., Lauro, L. J. R., & Bushman, B. J. (2019) · Brain stimulation · doi:10.1016/j.brs.2019.10.015 · Cited by 55
“Studies of the left hemisphere show that the lVLPFC is recruited especially in reappraisal-related emotion regulation strategies [16,31,32], and impulsivity control [29,30,33].”
BackgroundNeurocomputational mechanisms of reinforcement-guided learning in humans: A review
Cohen, M. X. (2008) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.8.2.113 · Cited by 55
“Significant correlations are often observed in the striatum and frontal cortex (Braver & Brown, 2003; Haruno et al., 2004; McClure, Berns, & Montague, 2003; O'Doherty, Dayan, Friston, Critchley, & Dolan, 2003; O'Doherty et al., 2007; Tanaka et al., 2004), and are interpreted as reflecting reward prediction error signa…”
BackgroundWillner, P., Bailey, R. J., Parry, R., & Dymond, S. (2010) · American Journal on Intellectual and Developmental Disabilities · doi:10.1352/1944-7558-115.2.157 · Cited by 56
Serotonergic regulation of the dopaminergic system: Implications for reward-related functions
Courtiol, E., Menezes, E. D. C., & Teixeira, C. M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.022 · Cited by 53
Working Together May Be Better: Activation of Reward Centers during a Cooperative Maze Task
Krill, A. L., & Platek, S. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0030613 · Cited by 42
Prospective Coding of Dorsal Raphe Reward Signals by the Orbitofrontal Cortex
Zhou, J., Jia, C., Feng, Q., Bao, J., & Luo, M. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.4017-14.2015 · Cited by 48
“Imaging, microdialysis, and pharmacological studies suggest that 5-HT mediates certain aspects of reward (Phillips et al., 1976; Redgrave and Horrell, 1976; Schwartz et al., 1990; Lorrain et al., 1997; Tanaka et al., 2004; Miyazaki et al., 2011b).”
BackgroundValuation, Intertemporal Choice, and Self-Control
Kable, J. W. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00010-3 · Cited by 66
“In this context, Tanaka and colleagues (Tanaka et al., 2004; reported the intriguing result that delay sensitivity varied across the striatum, with BOLD activity in ventromedial areas of the striatum reflecting a greater degree of discounting and BOLD activity in dorsolateral areas of the striatum…”
BackgroundResultInfluentialAnticipatory reward signals in ventral striatal neurons of behaving rats
Khamassi, M., Mulder, A. B., Tabuchi, E., Douchamps, V., & Wiener, S. I. (2008) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2008.06480.x · Cited by 57
“Interestingly, in a recent brain imaging study of humans performing a reward-motivated task, different striatal subregions were selectively active according to the discount factor that best modelled the subjects’ strategy concerning short- or long-term gain (Tanaka et al., 2004).”
BackgroundClassifying suicidal behavior with resting-state functional connectivity and structural neuroimaging
Gosnell, S. N., Fowler, J. C., & Salas, R. (2019) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13029 · Cited by 49
“Frontal alterations may contribute to dysfunctions in emotional pain processing (18) and executive control of mood (19), potentially associated with suicide.”
BackgroundCarlisi, C. O., Norman, L., Murphy, C. M., Christakou, A., Chantiluke, K., Giampietro, V., Simmons, A., Brammer, M., Murphy, D. G., MRC AIMS consortium, et al. (2017) · Psychological Medicine · doi:10.1017/s0033291717001088 · Cited by 49
“2009) and adults (Tanaka et al. 2004; Bickel et al. 2009; Hinvest et al. 2011), the present results highlight a differential abnormality in the posterior insula during reward presentation and internal state evaluation (Elliott et al.”
Petersen, K. J., van Wouwe, N. C., Stark, A. J., Lin, Y., Kang, H., Trujillo, P., KESSLER, R. M., Zald, D. H., Donahue, M. J., & Claassen, D. O. (2017) · Human Brain Mapping · doi:10.1002/hbm.23860 · Cited by 51
“…parametric mapping TE echo time TI inversion time TR repetition time r 510 r cortical networks are implicated in action choice and reward valuation [Bogacz and Gurney, 2007; Tanaka et al., 2004], we tested whether functional connectivity was linked to differences in incentive-based learning.”
BackgroundIntegration of Cognitive and Motivational Information in the Primate Lateral Prefrontal Cortex
SAKAGAMI, M., & WATANABE, M. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.010 · Cited by 48
Brooks, S. J. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0714-z · Cited by 45
“Additionally, BPI highlights the role of reinforcement learning in the brain and implicates the corticostriatal pathways in reward prediction [110], involving differential activation in this pathway for immediate versus delayed rewards [107].”
BackgroundMemory Deficits in Aging and Neurological Diseases
Khan, Z. U., Martín‐Montañez, E., Navarro‐Lobato, I., & Muly, E. C. (2014) · Progress in molecular biology and translational science · doi:10.1016/b978-0-12-420170-5.00001-5 · Cited by 58
A reinforcement learning model of joy, distress, hope and fear
Broekens, J., Jacobs, E., & Jonker, C. M. (2015) · Connection Science · doi:10.1080/09540091.2015.1031081 · Cited by 67
“…elements have been identified in the animal brain including the encoding of utility (Tanaka et al., 2004), changes in utility (Haruno & Kawato, 2006), and reward and motivational action value (Berridge, 2003; Berridge & Robinson, 2003; Berridge, Robinson, & Aldridge, 2009; Tanaka et al., 2004).”
BackgroundTemporal decision-making: insights from cognitive neuroscience
Luhmann, C. (2009) · Frontiers in Behavioral Neuroscience · doi:10.3389/neuro.08.039.2009 · Cited by 43
Nascimento, S. S., Oliveira, L. R., & DeSantana, J. M. (2018) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2018.06.006 · Cited by 46
Mechanisms of cerebellar involvement in associative learning
Bellebaum, C., & Daum, I. (2009) · Cortex · doi:10.1016/j.cortex.2009.07.016 · Cited by 51
Kobayashi, S., Kawagoe, R., Takikawa, Y., Koizumi, M., Sakagami, M., & Hikosaka, O. (2006) · Experimental Brain Research · doi:10.1007/s00221-006-0622-4 · Cited by 51
“This hypothesis is in agreement with recent fMRI studies that revealed different brain structures involved in reward prediction at different time scales (McClure et al. 2004; Tanaka et al. 2004 ).”
Koot, S., Baars, A., Hesseling, P., van den Bos, R., & Joëls, M. (2013) · Neuropharmacology · doi:10.1016/j.neuropharm.2013.02.008 · Cited by 46
Multi-timescale reinforcement learning in the brain
Masset, P., Tano, P., Kim, H. R., Malik, A. N., Pouget, A., & Uchida, N. (2025) · Nature · doi:10.1038/s41586-025-08929-9 · Cited by 58
Age-related prefrontal impairments implicate deficient prediction of future reward in older adults
Eppinger, B., Heekeren, H. R., & Li, S. (2015) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2015.04.010 · Cited by 49
Voon, V., & Dalley, J. W. (2015) · Current topics in behavioral neurosciences · doi:10.1007/7854_2015_5013 · Cited by 53
Kelly, E., Escamilla, C. O., & Tsai, P. T. (2020) · Neuroscience · doi:10.1016/j.neuroscience.2020.11.012 · Cited by 50
Muneoka, K. T., Shirayama, Y., Takigawa, M., & Shioda, S. (2008) · Neurochemical Research · doi:10.1007/s11064-008-9818-2 · Cited by 49
“In addition, a recent imaging study in humans suggested that the 5-HT system in the dlFC is involved in behavior in which future rewards are preferred over immediate rewards [94].”
BackgroundA New Face Image Recognition Algorithm Based on Cerebellum-Basal Ganglia Mechanism
Tang, S., & Shabaz, M. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/3688881 · Cited by 44
“Humans can recognize faces from a distance, even if facial details (such as eyes, nose, and mouth) are not visible. suggests that all the geometric characteristics of the face are sufficient for identification [18].”
BackgroundRetractedIntentional and Incidental Self-Control in Ventrolateral PFC
Cohen, J. R., Berkman, E. T., & Lieberman, M. D. (2011) · Cited by 46
Chakravarthy, V. S., Balasubramani, P. P., Mandali, A., Jahanshahi, M., & Moustafa, A. A. (2018) · Physiology & Behavior · doi:10.1016/j.physbeh.2018.06.032 · Cited by 47
Identification of animal behavioral strategies by inverse reinforcement learning
Yamaguchi, S., Naoki, H., Ikeda, M., Tsukada, Y., Nakano, S., Mori, I., & Ishii, S. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006122 · Cited by 61
Amygdala–prefrontal connectivity modulates loss aversion bias in anxious individuals
Xu, P., Dam, N. T. V., van Tol, M., Shen, X., Cui, Z., Gu, R., Qin, S., Aleman, A., Fan, J., & Luo, Y. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116957 · Cited by 48
Structural covariance of the neostriatum with regional gray matter volumes
Soriano‐Mas, C., Harrison, B. J., Pujol, J., López‐Solà, M., Hernández‐Ribas, R., Alonso, P., Contreras‐Rodríguez, O., Giménez, M., Blanco‐Hinojo, L., Ortiz, H., et al. (2012) · Brain Structure and Function · doi:10.1007/s00429-012-0422-5 · Cited by 43
“action and processing future rewards (Graybiel, 2008; Tanaka et al., 2004).”
BackgroundMapping memory function in the medial temporal lobe with the immediate-early gene Arc
Sauvage, M. M., Nakamura, N. H., & Beer, Z. (2013) · Behavioural Brain Research · doi:10.1016/j.bbr.2013.04.048 · Cited by 54
Barrós-Loscertales, A., Ventura-Campos, N., Sanjuán-Tomás, A., Belloch, V., Parcet, M.-A., & Ávila, C. (2010) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsq012 · Cited by 38
“…sensory stimuli (McClure et al., 2004), while its function has been related to the processing of secondary emotional stimuli (Kawabata and Zeki, 2004; Meseguer et al., 2007), reward prediction (Critchley et al., 2001; Tanaka et al., 2004) and the reward value of stimuli (Montague and Berns, 2002).”
BackgroundCortico-basal ganglia circuits involved in different motivation disorders in non-human primates
Sgambato, V., Worbe, Y., Epinat‐Duclos, J., Féger, J., & Tremblay, L. (2014) · Brain Structure and Function · doi:10.1007/s00429-014-0911-9 · Cited by 45
Impact of Size and Delay on Neural Activity in the Rat Limbic Corticostriatal System
Roesch, M. R., & Bryden, D. W. (2011) · Frontiers in Neuroscience · doi:10.3389/fnins.2011.00130 · Cited by 42
“Consistent with this notion, several labs have shown that VS incorporates expected value information into its neural firing (Bowman et al., 1996; Hassani et al., 2001; Carelli, 2002; Cromwell and Schultz, 2003; Setlow et al., 2003; Janak et al., 2004; Tanaka et al., 2004; Nicola, 2007; Khamassi et al., 2008; Ito and D…”
BackgroundInfluentialAltered functioning of reward circuitry in youth offspring of parents with bipolar disorder
Manelis, A., Ladouceur, C. D., Graur, S., Monk, K., Bonar, L. K., Hickey, M. B., Dwojak, A., Axelson, D., Goldstein, B., Goldstein, T. R., et al. (2015) · Psychological Medicine · doi:10.1017/s003329171500166x · Cited by 43
“…striatum (VS) supports reward anticipation and prediction error (Schultz et al. 2000; Knutson et al. 2001; Pagnoni et al. 2002; O’Doherty, 2004; Tanaka et al. 2004), the pallidum encodes expected reward value (Tachibana & Hikosaka, 2012), the amygdala, stimulus-value associations (Baxter &…”
BackgroundMethodResultInfluentialEngelmann, J. B., Maciuba, B., Vaughan, C., Paulus, M. P., & Dunlop, B. W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078292 · Cited by 46
Toward an understanding of the cerebral substrates of woman's orgasm
Bianchi-Demicheli, F., & Ortigue, S. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.04.016 · Cited by 48
“Similarly, the insula recently took a pivotal position in reward processing and especially in prediction of future reward (e.g., O’Doherty, 2004; Tanaka et al., 2004).”
BackgroundControlling instabilities in manipulation requires specific cortical-striatal-cerebellar networks
Mosier, K. M., Lau, C. C., Wang, Y., Venkadesan, M., & Valero‐Cuevas, F. J. (2011) · Journal of Neurophysiology · doi:10.1152/jn.00757.2010 · Cited by 40
Brain functional connectivity changes in children that differ in impulsivity temperamental trait
Inuggi, A., Sanz‐Arigita, E., González-Salinas, C., Valero-García, A. V., GarcÃa-Santos, J. M., & Fuentes, L. J. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00156 · Cited by 43
We are what we Consume: The Influence of Food Consumption on Impulsive Choice
Mishra, A., & Mishra, H. (2010) · Journal of Marketing Research · doi:10.1509/jmkr.47.6.1129 · Cited by 49
“Research suggests that impulsive people focus more on rewards than on punishments because their reward-seeking centers are more activated than those of other people (McClure et al. 2007; Tanaka et al. 2004).”
Functional coherence of insula networks is associated with externalizing behavior.
Abram, S. V., Wisner, K., Grazioplene, R. G., Krueger, R. F., MacDonald, A. W., & DeYoung, C. G. (2015) · Journal of Abnormal Psychology · doi:10.1037/abn0000078 · Cited by 43
Regulation of natural killer cell redistribution by prefrontal cortex during stochastic learning
Ohira, H., Fukuyama, S., Kimura, K., Nomura, M., Isowa, T., Ichikawa, N., Matsunaga, M., Shinoda, J., & Yamada, J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.04.088 · Cited by 42
“Some theorists have argued that the vagus nerve or parasympathetic nervous system may play a critical role in such flexible regulation (Thayer and Brosschot, 2005).”
BackgroundQuantized Attention-Gated Kernel Reinforcement Learning for Brain–Machine Interface Decoding
Wang, F., Wang, Y., Xu, K., Li, H., Liao, Y., Zhang, Q., Zhang, S., Zheng, X., & Príncipe, J. C. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2493079 · Cited by 39
“Mahmoudi and Sanchez [18] further developed the work with an internal reward represented by neural activity in the nucleus accumbens [19], [20].”
BackgroundArnautovska, U., Kesby, J. P., Korman, N., Rebar, A. L., Chapman, J., Warren, N., Rossell, S. L., Dark, F. L., & Siskind, D. J. (2022) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s393775 · Cited by 40
“Importantly, animal and human studies have shown that activation of the ventral tegmental area from hedonic stimuli, such as food and social connection, results in phasic dopamine release to the nucleus accumbens.(39,40) Therefore, motivation for PA may also be increased through social and hedonic experiences (eg, enj…”
BackgroundMultistage Valuation Signals and Common Neural Currencies
Platt, M. L., & Plaßmann, H. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00013-9 · Cited by 41
Funciones de la corteza prefrontal ventromedial en la toma de decisiones emocionales
Contreras, D., Catena, A., Cándido, A., Perales, J. C., & Maldonado, A. (2008) · Redalyc (Universidad Autónoma del Estado de México) · Cited by 75
Neural signatures of intransitive preferences
Kalenscher, T. (2010) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2010.00049 · Cited by 36
“…revealed activation in right posterior insula which has been previously associated with choices of larger, but riskier (Huettel et al., 2006; Smith et al., 2009; Xue et al., 2009) or larger, but delayed rewards (Tanaka et al., 2004; Wittmann et al., 2007) (Table S6 in Supplementary Material).”
InfluentialThe Neural Correlates of Timing Functions
Rubia, K. (2006) · WORLD SCIENTIFIC eBooks · doi:10.1142/9789812707123_0009 · Cited by 42
Valuation of uncertain and delayed rewards in primate prefrontal cortex
Kim, S., Hwang, J., Seo, H., & Lee, D. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.010 · Cited by 37
“…Berns, Laibson, & Loewenstein, 2007; Gregorios-Pippas, Tobler, & Schultz, 2009; Kable & Glimcher, 2007; Luhmann, Chun, Yi, Lee, & Wang, 2008; McClure, Ericson, Laibson, Loewenstein, & Cohen, 2007; McClure, Laibson, Loewenstein, & Cohen, 2004; Tanaka et al., 2004; Weber & Huettel, 2008).”
Hierarchical gradients of multiple timescales in the mammalian forebrain
Song, M. J., Shin, E. J., Seo, H., Soltani, A., Steinmetz, N. A., Lee, D., Jung, M. W., & Paik, S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415695121 · Cited by 45
“During reinforcement learning, learning rates may adjust based on the stability of reward probabilities ( 56 – 58 ), and multiple discount factors may operate concurrently to estimate the values of immediate versus delayed rewards ( 59 ).”
Integrative Networks Across Basal Ganglia Circuits
Haber, S. N. (2010) · Handbook of behavioral neuroscience · doi:10.1016/b978-0-12-374767-9.00024-x · Cited by 42
Advances in techniques for imposing reciprocity in brain-behavior relations
Turner, B. M., Palestro, J. J., Miletić, S., & Forstmann, B. U. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.04.018 · Cited by 37
When Does Reward Maximization Lead to Matching Law?
Sakai, Y., & Fukai, T. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003795 · Cited by 42
Estimation and Testing of Computational Psychological Models
Busemeyer, J. R., & Diederich, A. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00004-8 · Cited by 35
The emergence of saliency and novelty responses from Reinforcement Learning principles
Laurent, P. A. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.09.004 · Cited by 36
Palomo, T., Beninger, R. J., Kostrzewa, R. M., & Archer, T. (2008) · Neurotoxicity Research · doi:10.1007/bf03033572 · Cited by 36
“…such as the lateral prefrontal area and posterior parietal cortex, contributing for example to economical and rational planning, and time-reward trade-offs, offer different competing decision-making networks is also plausible (McClure et al., 2004; Tanaka et al., 2004; Sanfey et al., 2006).”
BackgroundYeomans, M. R., & Brace, A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0137626 · Cited by 37
“How then might exposure to food cues enhance impulsive responding on the DDT and risk-taking on the BART? The present data cannot offer a mechanistic explanation but other data suggest at least two possibilities based around evidence that two key brain areas, the striatum and orbitofrontal cortex (OFC), are implicated…”
BackgroundThe neural dissociation of subjective valuation from choice processes in intertemporal choice
Liu, L., Feng, T., Wang, J., & Li, H. (2012) · Behavioural Brain Research · doi:10.1016/j.bbr.2012.02.045 · Cited by 42
Sellitto, M., Ciaramelli, E., Mattioli, F., & di Pellegrino, G. (2016) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2015.00367 · Cited by 35
“…associated insula activity with choice of either delayed (Wittmann et al., 2007; Claus et al., 2011; Kayser et al., 2012; Liu and Feng, 2012; Luo et al., 2012) or immediate options (Tanaka et al., 2004; McClure et al., 2007; Wittmann et al., 2010), and therefore its role in TD behavior is unclear.”
BackgroundA Neural Correlate of Motivational Conflict in the Superior Colliculus of the Macaque
Isoda, M., & Hikosaka, O. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90275.2008 · Cited by 37
Interest in politics modulates neural activity in the amygdala and ventral striatum
Gozzi, M., Zamboni, G., Krüeger, F., & Grafman, J. H. (2010) · Human Brain Mapping · doi:10.1002/hbm.20976 · Cited by 39
“Activity in the striatum was found in response to food, sex, drugs and money [Moll et al., 2005; O’Doherty et al., 2002; Schultz et al., 1997; Tanaka et al., 2004].”
Background
Metadata
- Authors
- Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Japan Science and Technology Agency (JP)
- Hiroshima University (JP)
- Corresponding author
- Doya, Kenji
- Venue
- Nature Neuroscience
- Published
- 2004-07-04
- Volume
- 7
- Issue
- 8
- Pages
- 887-893
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2032610016
- Cited by
- 779
- References
- 54
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 9 · 2022: 29 · 2021: 28 · 2020: 32 · 2019: 25 · 2018: 32 · 2017: 37 · 2016: 40 · 2015: 45 · 2014: 50 · 2013: 42 · 2012: 47
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- reward prediction, time scales, cortico-basal ganglia loop, prediction error, insula, striatum, orbitofrontal cortex, dorsolateral prefrontal cortex, dorsal raphe nucleus, cerebellum, computational modeling, reinforcement learning
- MeSH terms
- Adult, Brain, Brain Mapping, Discrimination Learning, Female, Humans, Magnetic Resonance Imaging, Male, Markov Chains, Middle Aged, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.