Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Evaluation of both immediate and future outcomes of one's actions is a critical requirement for intelligent behavior. Using functional magnetic resonance imaging (fMRI), we investigated brain mechanisms for reward prediction at different time scales in a Markov decision task. When human subjects learned actions on the basis of immediate rewards, significant activity was seen in the lateral orbitofrontal cortex and the striatum. When subjects learned to act in order to obtain large future rewards while incurring small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects' performance data revealed graded maps of time scale within the insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior regions were involved in predicting future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In neurotypical individuals, the key brain loci that have been implicated in prediction are basal ganglia (68, 69), striatum (70, 71), anterior cingulate (72), and cerebellum (73, 74).”
From Autism as a disorder of prediction (2014) · cites this paper as [69]
Citations
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“, which has been successfully used for explaining reward-predictive activations of the midbrain dopaminergic system as well as those of the cortex and the striatu”
MethodEmotion, Decision Making and the Orbitofrontal Cortex
Bechara, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.295 · Cited by 3,589 · cited 1×
“Damage to the prefrontal cortex often impairs daily decision making, which requires assessment of future outcome”
BackgroundBarto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Basal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“The cortex and the basal ganglia both have parallel loop organization, with four major loops (limbic, cognitive, motor and oculomotor) and finer, topographic sub-loops within each major loo”
Background(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
The Orbitofrontal Cortex and Reward
Rolls, E. T. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.284 · Cited by 1,595
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001) · Neuron · doi:10.1016/s0896-6273(01)00303-8 · Cited by 1,404
Cited by
Items 501–600 of 798
Marutani, T., Yahata, N., Ikeda, Y., Ito, T., Yamamoto, M., Matsuura, M., Matsushima, E., Okubo, Y., Suzuki, H., & Matsuda, T. (2011) · Psychiatry and Clinical Neurosciences · doi:10.1111/j.1440-1819.2011.02189.x · Cited by 16
Varma, M. M., Zhen, S., & Yu, R. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.120363 · Cited by 18
“Subsequent studies have supported similar conclusions regarding the involvement of different neural circuits and regions during immediate and delayed reward decision-making (e.g. Christakou et al., 2011; McClure et al., 2007; Tanaka et al., 2004).”
BackgroundResultAre Risky Choices Actually Guided by a Compensatory Process? New Insights from fMRI
Rao, L., Zhou, Y., Xu, L., Liang, Z., Jiang, T., & Li, S. (2011) · PLoS ONE · doi:10.1371/journal.pone.0014756 · Cited by 16
van den Bos, R. (2014) · Behavioural Pharmacology · doi:10.1097/fbp.0000000000000110 · Cited by 18
“…bias the balance between the dorsal striatum and the ventral striatum: high levels of this neurotransmitter tilt the balance towards the dorsal striatum, whereas low levels tilt the balance towards the ventral striatum [Tanaka et al. (2004, 2007); for review see van den Bos et al. (2013b, 2014a)].”
BackgroundAinslie, G. W. (2010) · The MIT Press eBooks · doi:10.7551/mitpress/9780262513111.003.0009 · Cited by 20
“Furthermore, even within β centers delay sensitivity is not uniform, but can be pinpointed in the striatum by a graded map with ventroanterior regions tracking more immediate rewards and dorsoposterior regions tracking more delayed ones (Tanaka et al. 2004).”
BackgroundCircadian Plasticity of Mammalian Inhibitory Interneurons
Jasińska, M., & Pyza, E. (2017) · Neural Plasticity · doi:10.1155/2017/6373412 · Cited by 17
Mood dysregulation and stabilization: perspectives from emotional cognitive neuroscience
Yamawaki, S., Okada, G., Okamoto, Y., & Liberzon, I. (2011) · The International Journal of Neuropsychopharmacology · doi:10.1017/s1461145711000757 · Cited by 17
“We thank our colleagues and collaborators including Shinpei Yoshimura, Atuo Yoshino, Keiichi Onoda, Kazutaka Ueda, Saori C. Tanaka, Nicolas Schweighofer, and Kenji Doya, for their critical contributions to the work described in this review.”
BackgroundSelf-citationMatsui, T., Hattori, Y., Tsumura, K., Aoki, R., Takeda, M., Nakahara, K., & Jimura, K. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.118892 · Cited by 18
Septotemporal variations in hippocampal value and outcome processing
Yun, M., Hwang, J. Y., & Jung, M. W. (2023) · Cell Reports · doi:10.1016/j.celrep.2023.112094 · Cited by 16
Encoding of eye movements explains reward-related activity in cerebellar simple spikes
Lixenberg, A., Yarkoni, M., Botschko, Y., & Joshua, M. (2020) · Journal of Neurophysiology · doi:10.1152/jn.00363.2019 · Cited by 15
Reward and plasticity: Implications for neurorehabilitation
Johnson, B. P., & Cohen, L. G. (2022) · Handbook of clinical neurology · doi:10.1016/b978-0-12-819410-2.00018-7 · Cited by 16
Castellanos, F. X. (2008) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.11.009 · Cited by 19
Neural network integration during the perception of in-group and out-group members
Greven, I. M., & Ramsey, R. (2017) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.09.036 · Cited by 16
Coffman, B. A., Torrence, N., Murphy, T. K., Bebko, G. M., Graur, S., Chase, H. W., Salisbury, D. F., & Phillips, M. L. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.05.029 · Cited by 15
Remembering rewarding futures: A simulation‐selection model of the hippocampus
Jung, M. W., Lee, H., Jeong, Y., Lee, J. W., & Lee, I. (2018) · Hippocampus · doi:10.1002/hipo.23023 · Cited by 18
The Neural Basis of Motivational Influences on Cognitive ControlPreprint
Parro, C., Dixon, M. L., & Christoff, K. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/113126 · Cited by 16
Decoding reward–curiosity conflict in decision-making from irrational behaviors
Konaka, Y., & Naoki, H. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00439-w · Cited by 15
Bi, C., & Qi, H. (2022) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2022.01106 · Cited by 14
Reward-Based Learning and Emotional Habit Formation in the Cerebellum
Pierce, J. E., & Péron, J. (2022) · Advances in experimental medicine and biology · doi:10.1007/978-3-030-99550-8_9 · Cited by 18
Do Basal Ganglia Amplify Willed Action by Stochastic Resonance? A Model
Chakravarthy, V. S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0075657 · Cited by 15
How the Experience of Time Shapes Decision-Making
Wittmann, M., & Paulus, M. P. (2016) · Studies in neuroscience, psychology and behavioral economics · doi:10.1007/978-3-642-35923-1_8 · Cited by 18
Modeling sensory-motor decisions in natural behavior
Zhang, R., Zhang, S., Tong, M. H., Cui, Y., Rothkopf, C. A., Ballard, D. H., & Hayhoe, M. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006518 · Cited by 15
“92 An fMRI study by [16] suggests that different cortico-basal ganglia loops are responsible 93 for reward prediction at different time scales, allowing multiple discount factors to be 94 implemented.”
BackgroundNeural Basis of Professional Pride in the Reaction to Uniform Wear
Hong, Y.-J., Park, S., Kyeong, S.-H., & Kim, J. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00253 · Cited by 15
“The imaging results showing the interaction of individual and clothes included the insula, which is part of the empathy network (Engen and Singer, 2013) and is also involved in emotion regulation and reward processing (Tanaka et al., 2004; Villafuerte et al., 2012).”
BackgroundDemoto, Y., Okada, G., Okamoto, Y., Kunisato, Y., Aoyama, S., Onoda, K., Munakata, A., Nomura, M., Tanaka, S., Schweighofer, N., et al. (2012) · Neuropsychobiology · doi:10.1159/000328990 · Cited by 15
Self-citationTrial-to-trial Adaptation: Parsing out the Roles of Cerebellum and BG in Predictive Motor Timing
Lungu, O., Bareš, M., Liu, T., Gomez, C. M., Cechova, I., & Ashe, J. (2016) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00943 · Cited by 16
“The two structures may have a fundamentally different relation to reward during learning with the striatum being activated primarily for immediate rewards whereas the cerebellum is involved with future rewards (Tanaka et al., 2004).”
BackgroundHaber, S. N. (2009) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195373035.003.0004 · Cited by 14
Khamassi, M., Wilson, C. R. E., Rothé, M., Quilodran, R., Dominey, P. F., & Procyk, E. (2011) · The MIT Press eBooks · doi:10.7551/mitpress/9780262016438.003.0019 · Cited by 14
Reward (BIS/BAS) mechanisms and fatigue in patients on chronic hemodialysis
Bossola, M., Angioletti, L., Stasio, E. D., Vulpio, C., Filippis, D. D., & Balconi, M. (2019) · Psychology Health & Medicine · doi:10.1080/13548506.2019.1653477 · Cited by 14
“be derived from an alteration of the function of the basal ganglia secondary to inflammation (Dobryakova et al., 2017; Tanaka et al., 2004).”
BackgroundBiases and optimality of sensory-motor and cognitive decisions
Trommershäuser, J. (2009) · Progress in brain research · doi:10.1016/s0079-6123(09)01321-1 · Cited by 16
Correlates of reward‐predictive value in learning‐related hippocampal neural activity
Okatan, M. (2009) · Hippocampus · doi:10.1002/hipo.20535 · Cited by 15
“Other brain areas that show learning-related activity correlated with reward predictive value may also play a role in this process (Liu and Richmond, 2000; Schultz, 2000; Barra-clough et al., 2004; Seymour et al., 2004; Tanaka et al., 2004; Samejima et al., 2005; O’Doherty et al., 2006).”
InfluentialAmita, H., & Matsushima, T. (2014) · Behavioural Brain Research · doi:10.1016/j.bbr.2014.04.004 · Cited by 17
“Choice impulsiveness has been a topic of intensive experimental research in a wide range of behavioral sciences, including psychology [1, 2], psychopharmacology [3-6], behavioral ecology [7-10] and neuroscience [11-16].”
Bennett, M. S. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.685853 · Cited by 14
“For example, the dorsolateral prefrontal cortex activates when considering future rewards, but not when considering present rewards (McClure, 2004; Tanaka et al., 2004; Kim et al., 2008).”
BackgroundConceptual Representation and the Making of New Decisions
Boorman, E. D., & Rushworth, M. F. S. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.014 · Cited by 15
“In such models the vmPFC is assigned a preeminent role because activity in this region is often correlated with the value Neuron 63, of the choice that is being taken by a subject (Tanaka et al., 2004).”
BackgroundOn the nature, modeling, and neural bases of social ties
van Winden, F. A. A. M., Stallen, M., & Ridderinkhof, K. R. (2008) · Advances in health economics and health services research · doi:10.1016/s0731-2199(08)20006-3 · Cited by 22
Balasubramani, P. P., & Chakravarthy, V. S. (2019) · Cognitive Neurodynamics · doi:10.1007/s11571-019-09564-7 · Cited by 14
Zimmerman, A., Ferriday, D., Davies, S. R., Martin, A., Rogers, P. J., Mason, A., & Brunstrom, J. M. (2017) · Appetite · doi:10.1016/j.appet.2017.05.019 · Cited by 18
Neuronal Representation of the Saccadic Timing Signals in Macaque Lateral Intraparietal Area
Zhou, Y., Liu, Y., Wu, S., & Zhang, M. (2017) · Cerebral Cortex · doi:10.1093/cercor/bhx166 · Cited by 13
Okamoto, Y. (2005) · Folia Pharmacologica Japonica · doi:10.1254/fpj.126.194 · Cited by 13
Self-citationMonetary reward magnitude effects on behavior and brain function during goal-directed behavior
Rosell‐Negre, P., Bustamante, J. C., Fuentes‐Claramonte, P., Costumero, V., Benabarre, S., & Barrós‐Loscertales, A. (2016) · Brain Imaging and Behavior · doi:10.1007/s11682-016-9577-7 · Cited by 13
Wei, Z., Han, L., Zhong, X., Liu, Y., Zha, R., Wang, Y., Yang, L., Bu, J., Song, H., Wang, W., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.11.048 · Cited by 13
Dynamical Motor Control Learned with Deep Deterministic Policy Gradient
Shi, H., Sun, Y., & Li, J. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/8535429 · Cited by 12
“Themodel proposed in this study has a loose resemblance to the specializations of the biological nervous system, where the DDPG learning corresponds to the rewarding circuitry [20, 21], and the RNN actor corresponds to the cortical motor areas.”
BackgroundIntegrative Networks Across Basal Ganglia Circuits
Haber, S. N. (2016) · Handbook of behavioral neuroscience · doi:10.1016/b978-0-12-802206-1.00027-1 · Cited by 18
Rodriguez‐Moreno, D. V., Cycowicz, Y. M., Figner, B., Wang, Z., He, X., Geronazzo‐Alman, L., Sun, X., Cheslack‐Postava, K., Bisaga, A. M., Hoven, C. W., et al. (2021) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2021.100942 · Cited by 13
“…subjective value of a reward (Carter, 2010; Hare, O’Doherty, Camerer, Schultz, & Rangel, 2008; Juechems et al., 2017; Kable and Glimcher, 2007; McClure et al., 2004a, 2004b; Tanaka et al., 2004), and the VS as associated with valuation of delayed rewards (Hare et al., 2014; Prévost et al., 2010).”
BackgroundSon, Y., Kim, Y.-B., Kim, J., Kim, J.-H., Kwon, D.-H., Lee, H., & Cho, Z. (2022) · Pharmaceuticals · doi:10.3390/ph15050583 · Cited by 12
“Based on previous findings that the striatum and corticobasal ganglia pathways play an important role in the neuropathology of affective disorders and subsequent suicidal behaviors [91] and are associated with reward prediction involved in decision-making [92], one study suggested that the reduced availability of SERT…”
ResultModeling Decision-Making Systems in Addiction
Kurth-Nelson, Z. L., & Redish, A. D. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0751-5_6 · Cited by 14
“There is evidence that a range of discounting factors are calculated in the striatum, with a gradient from faster discount rates represented in ventral striatum to slower rates in dorsal striatum (Tanaka et al. 2004).”
BackgroundFrom reward value to decision-making: neuronal and computational principles
Rolls, E. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374620-7.00005-4 · Cited by 17
Panksepp, J., & Wilson, C. G. (2016) · Studies in neuroscience, psychology and behavioral economics · doi:10.1007/978-3-642-35923-1_13 · Cited by 14
Angioletti, L., Bossola, M., Filippis, D. D., & Balconi, M. (2018) · Health Psychology Open · doi:10.1177/2055102918809761 · Cited by 14
“Reward system was previously hypothesized as involved in the functioning of clinical populations with chronic disease (Dantzer et al., 2015; Dobryakova et al., 2017; Tanaka et al., 2004).”
BackgroundChapter 2 Neurochemistry of cognition: serotonergic and adrenergic mechanisms
Ph, R., & Benoît, M. (2008) · Handbook of clinical neurology · doi:10.1016/s0072-9752(07)88002-x · Cited by 13
Hamid, A. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.07.005 · Cited by 11
Hasuzawa, S., Tomiyama, H., Murayama, K., Ohno, A., Kang, M., Mizobe, T., Kato, K., Matsuo, A., Kikuchi, K., Togao, O., et al. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.836965 · Cited by 11
Reward and Neurocomputational Processes
Frank, G. (2010) · Current topics in behavioral neurosciences · doi:10.1007/7854_2010_81 · Cited by 11
“Most recently, immediately rewarding processes have been related to ventral striatal activation, whereas delayed gratification may activate dorsal striatal brain response (Tanaka, Doya et al. 2004).”
BackgroundMethodImpulsiveness and Inhibitory Mechanisms
Bari, A., Kellermann, T. S., & Studer, B. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-800935-2.00006-3 · Cited by 13
Zhang, L., van der Meer, L., Opmeer, E. M., Marsman, J.-B. C., Ruhé, H. G., & Aleman, A. (2016) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2016.09.020 · Cited by 11
The Basal Ganglia and the Encoding of Value
Doya, K., & Kimura, M. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00026-9 · Cited by 11
Self-citationBellot, J., Sigaud, O., & Khamassi, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33093-3_29 · Cited by 11
Decreased Basal Ganglia Volume in Cerebral Amyloid Angiopathy
Fotiadis, P., Pasi, M., Charidimou, A., Warren, A. D., Schwab, K. M., Initiative, A. D. N., Rosand, J. M., van der Grond, J., van Buchem, M. A., Viswanathan, A., et al. (2021) · Journal of Stroke · doi:10.5853/jos.2020.04280 · Cited by 11
Striatal and Pallidal Activation during Reward Modulated Movement Using a Translational Paradigm
Bischoff‐Grethe, A., Buxton, R. B., Paulus, M. P., Fleisher, A., Yang, T. T., & Brown, G. G. (2015) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617715000491 · Cited by 11
“…the response is a brief, ballistic button press (Elliott, Friston, & Dolan, 2000; Ernst et al., 2004; Knutson, Westdorp, Kaiser, & Hommer, 2000; Tanaka et al., 2004), whereas in animal studies the response is often a capture movement that engages both proximal and distal musculature (Tremblay,…”
BackgroundNakagawa, Y. (2017) · Journal of Drug and Alcohol Research · doi:10.4303/jdar/236017 · Cited by 10
Vassiliadis, P., Derosière, G., & Duqué, J. (2019) · eNeuro · doi:10.1523/eneuro.0458-18.2019 · Cited by 11
“…neuroimaging studies in humans have reported activity related to RPEs in the cerebellum (O’Doherty, 2004; Ramnani et al., 2004; Seymour et al., 2004; Tanaka et al., 2004; Tobler et al., 2006; Garrison et al., 2013), suggesting that this structure is functionally involved in processing reward…”
BackgroundSexual motivation is reflected by stimulus-dependent motor cortex excitability
Schecklmann, M., Engelhardt, K., Konzok, J., Rupprecht, R., Greenlee, M. W., Mokros, A., Langguth, B., & Poeppl, T. B. (2015) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsu157 · Cited by 10
“In this context, motivation directs thoughts and behavior to a sexual goal, including the perceived urge to express sexual behavior (Stoléru, 2014), and is mediated by cortico-basal ganglia loops (Tanaka et al., 2004; Kühn and Gallinat, 2011; Stoléru, 2014).”
Backgroundvan de Vijver, I., Ridderinkhof, K. R., Harsay, H. A., Reneman, L., Cavanagh, J. F., Buitenweg, J. I. V., & Cohen, M. X. (2016) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2016.06.002 · Cited by 10
Why Losses Loom Larger than Gains: Modeling Neural Mechanisms of Cognitive-Affective Interaction
Eliasmith, C., Litt, A., & Thagard, P. R. (2006) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 16
Arbitration between controlled and impulsive choices
Economides, M., Guitart‐Masip, M., Kurth-Nelson, Z. L., & Dolan, R. J. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.071 · Cited by 13
“It has been proposed that self-control involves a conflict between competing value systems (Hofmann et al., 2009; McClure et al., 2004; Tanaka et al., 2004), and this idea gains support from evidence that the brain draws on multiple systems when making decisions (Balleine, 2005; Daw et al., 2005;…”
BackgroundInfluentialZhang, P., Chao, L., Chen, Y., Ma, X., Wang, W., He, J., Huang, J., & Li, Q. (2020) · Sensors · doi:10.3390/s20195528 · Cited by 9
“[38,39] proved that the reward information in the autonomous RLBMI can be represented by the neural activities from the nucleus accumbens, and Marsh et al.”
MethodKalhan, S., Garrido, M. I., Hester, R., & Redish, A. D. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.09.032 · Cited by 10
Listos, J., Baranowska‐Bosiacka, I., Wąsik, A., Talarek, S., Tarnowski, M., Listos, P., Łupina, M., Antkiewicz‐Michaluk, L., Gutowska, I., Tkacz, M., et al. (2016) · Psychopharmacology · doi:10.1007/s00213-016-4289-7 · Cited by 10
“It is strongly involved in various aspects of addiction status (Delgado et al. 2003; Everitt and Robbins 2005; Tanaka et al. 2004).”
Enomoto, K., Matsumoto, N., Inokawa, H., Kimura, M., & Yamada, H. (2020) · Scientific Reports · doi:10.1038/s41598-020-65914-0 · Cited by 11
Multiscale Computation and Dynamic Attention in Biological and Artificial Intelligence
Badman, R. P., Hills, T. T., & Akaishi, R. (2020) · Brain Sciences · doi:10.3390/brainsci10060396 · Cited by 13
A One-Shot Learning Signal in Monkey Prefrontal Cortex
Achterberg, J., Kadohisa, M., Watanabe, K., Kusunoki, M., Buckley, M. J., & Duncan, J. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.1338-21.2021 · Cited by 11
Frontal operculum temporal difference signals and social motor response learning
Kumar, P., Waiter, G. D., Ahearn, T. S., Milders, M. M., Reid, I., & Steele, J. D. (2008) · Human Brain Mapping · doi:10.1002/hbm.20611 · Cited by 10
“…to DA activity [McClure et al., 2003a; Montague et al., 1996; O’Doherty et al., 2004; Paulus et al., 2004; Schultz, 2002; Seymour et al., 2007; Tanaka et al., 2004; Tobler et al., 2006; Waelti et al., 2001], and DA system manipulations in humans alter the TD signal [Menon et al., 2007;…”
BackgroundKohyama, J. (2012) · Journal of Behavioral and Brain Science · doi:10.4236/jbbs.2012.24055 · Cited by 9
“[30].”
Neural correlates in intertemporal choice of gains and losses.
Faralla, V., Benuzzi, F., Lui, F., Baraldi, P., Dimitri, N., & Nichelli, P. (2015) · Journal of Neuroscience Psychology and Economics · doi:10.1037/npe0000032 · Cited by 9
Colombo, B., Iannello, P., Puglisi, G., & Antonietti, A. (2020) · Studia Psychologica · doi:10.31577/sp.2020.02.795 · Cited by 9
“…several interacting neural systems are involved in the process of making intertemporal choices (McClure, Laibson, Loewenstein, & Cohen, 2004; McClure, Marzilli Ericson, Laibson, Loewenstein, & Cohen, 2007; Tanaka et al., 2004; Xu, Liang, Wang, Li, & Jiang, 2009; Wittmann, Leland, & Paulus, 2007).”
Altered value-based decision-making in anorexia nervosa: A systematic review
Brown, C. S., Nuñez, A., & Wierenga, C. E. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105944 · Cited by 8
“…increases in tandem with increasing subjective value of food stimuli in healthy participants (Plassmann et al., 2007) and is positively correlated with cue or action valuation as estimated using reinforcement learning models (Doya et al., 2004; Gl ¨ ascher et al., 2009; Wunderlich et al., 2010).”
Fatigue and apathy in patients on chronic hemodialysis
Bossola, M., Stasio, E. D., Monteburini, T., Santarelli, S., Cenerelli, S., Manes, M., Parodi, E. L., Angioletti, L., & Balconi, M. (2021) · Therapeutic Apheresis and Dialysis · doi:10.1111/1744-9987.13784 · Cited by 8
Large-scale societal dynamics are reflected in human mood and brain
Lebedev, A. V., Abé, C., Acar, K., Deco, G., Kringelbach, M. L., Ingvar, M., & Petrović, P. (2022) · Scientific Reports · doi:10.1038/s41598-022-08569-3 · Cited by 9
The Fast Spiking Subpopulation of Striatal Neurons Coding for Temporal Cognition of Movements
Shen, B., Wang, Z., & Wang, X. (2017) · Frontiers in Cellular Neuroscience · doi:10.3389/fncel.2017.00406 · Cited by 8
“Human fMRI (Tanaka et al., 2004) and animal in vivo electrophysiology studies (Matell et al., 2003; Mello et al., 2015) have revealed that striatal neurons are activated when a subject is predicting a reward or action, and they sustain modulation of their firing rate until the time the action is…”
BackgroundSingle-Neuron Correlates of Decision Confidence in the Human Medial Temporal Lobe
Unruh-Pinheiro, A., Hill, M. R., Weber, B., Boström, J. P., Elger, C. E., & Mormann, F. (2020) · Current Biology · doi:10.1016/j.cub.2020.09.021 · Cited by 9
Li, X., Hu, P., & Liu, J. (2019) · Journal of Pacific Rim Psychology · doi:10.1017/prp.2019.22 · Cited by 8
“…McClure et al., 2004; Peters & Büchel, 2009; Weber & Huettel, 2008; Wittmann, Leland, & Paulus, 2007), and the dorsolateral prefrontal cortex (Leon & Shadlen, 1999; Roesch & Olson, 2003; Schultz, Tremblay, & Hollerman, 2000; Tanaka et al., 2004; Tsujimoto & Sawaguchi, 2005; Wallis & Miller, 2003).”
Time preference and personal value: a population-based cross-sectional study in Japan
Kawakami, N., Watanabe, K., Nishi, D., Takagi, D., Hashimoto, H., & Tanaka, S. (2020) · BMC Psychology · doi:10.1186/s40359-020-00458-6 · Cited by 8
“Previous studies indicate that time discounting and hyperbolic discounting may stem from a combined activity between different neural networks in brain [21, 24, 26].”
BackgroundSelf-citationSimulation of Human Episodic Memory by Using a Computational Model of the Hippocampus
Sato, N., & Yamaguchi, Y. (2010) · Advances in Artificial Intelligence · doi:10.1155/2010/392868 · Cited by 8
“[64] applied the temporal difference (TD) learning algorithm to the BOLD signal analysis and detected a topographical map of time scales of reward predictions.”
MethodThe Motor, Cognitive, Affective, and Autonomic Functions of the Basal Ganglia
Moustafa, A. A., Mandali, A., Balasubramani, P. P., & Chakravarthy, V. S. (2018) · Cognitive science and technology · doi:10.1007/978-981-10-8494-2_3 · Cited by 8
Aeberhard, A., Gschwind, L., Kossowsky, J., Lukšys, G., Papassotiropoulos, A., de Quervain, D. J. ‐., & Vogler, C. (2018) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-018-0071-5 · Cited by 8
“For example, reinforcement learning can model behavior in a number of tasks where rewards or punishments of some kind (sometimes implicit) are involved, and despite different formalizations, most of these models have common parameters such as the learning rate, exploration-exploitation tradeoff, and future discounting…”
BackgroundIshida, F., Sasaki, T., Sakaguchi, Y., & Shimai, H. (2008) · Neurocomputing · doi:10.1016/j.neucom.2008.04.009 · Cited by 9
Uchibe, E., & Doya, K. (2005) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2005.1490949 · Cited by 8
“Recent neuroscience research revealed that there are parallel reinforcement learning pathways in the human brain, each specialized for reward prediction at different time scales [ 5 ].”
BackgroundSelf-citationA computational neural model of goal-directed utterance selection
Klein, M., Kamp, H., Palm, G., & Doya, K. (2010) · Neural Networks · doi:10.1016/j.neunet.2010.01.003 · Cited by 7
Self-citationInfluences of Reinforcement and Choice Histories on Choice Behavior in Actor-Critic Learning
Katahira, K., & Kimura, K. (2022) · Computational Brain & Behavior · doi:10.1007/s42113-022-00145-2 · Cited by 8
“Although much research in psychology and neuroscience has been conducted using RL models without state variables, several studies employing RL model-based analyses have used RL models that incorporate state variables (e.g., Tanaka et al. 2004; Sch-weighofer et al. 2008; Spiegler et al. 2020).”
MethodHigh Beta Power in the Ventrolateral Prefrontal Cortex Indexes Human Approach Behavior: A Case Study
Provenza, N. R., Rajesh, S. V., Reyes, G., Katlowitz, K. A., Pugalenthi, L. S., Bechtold, R. A., Diab, N., Reddy, S. D., Allam, A. K., Gandhi, A. D., et al. (2025) · Journal of Neuroscience · doi:10.1523/jneurosci.1321-24.2025 · Cited by 12
“The left vlPFC/anterior insula (along with striatum) is implicated in immediate future reward expectancy (Tanaka et al., 2004).”
Pleasure and Aversion: Challenging the Conventional Dichotomy
Ainslie, G. W. (2009) · Inquiry · doi:10.1080/00201740903087342 · Cited by 8
Funamizu, A., Ito, M., Doya, K., Kanzaki, R., & Takahashi, H. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00027 · Cited by 8
“Many anatomical and functional studies suggest that this diverse set of functions is simultaneously implemented in parallel in the circuit [anatomy: (Haber, 2003; Voorn et al., 2004; Gruber and McDonald, 2012); function: (Tanaka et al., 2004; Balleine et al., 2007; Yamin et al., 2013)].”
BackgroundInfluentialSelf-citationOscillation/Coincidence-Detection Models of Reward-Related Timing in Corticostriatal Circuits
Yin, B., Shi, Z., Wang, Y., & Meck, W. H. (2022) · Timing & Time Perception · doi:10.1163/22134468-bja10057 · Cited by 8
Decomposing brain signals involved in value-based decision making
Dreher, J. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374620-7.00006-6 · Cited by 8
Amplitude of low-frequency fluctuations in happiness: A resting-state fMRI study
Liu, J., Huang, X., Bi, C., Luo, Y., & BaoLin, L. I. (2015) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/n972014-00700 · Cited by 9
Khamassi, M. (2007) · HAL · doi:10.13140/rg.2.1.4065.8004 · Cited by 10
Independent and Interacting Value Systems for Reward and Information in the Human BrainPreprint
Dezza, I. C., Cleeremens, A., & Alexander, W. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.05.04.075739 · Cited by 8
Sequential Density Ratio Estimation for Simultaneous Optimization of Speed and AccuracyPreprint
Ebihara, A. F., Miyagawa, T., Sakurai, K., & Imaoka, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05587 · Cited by 8
The cognitive reality monitoring network and theories of consciousness
Cortese, A., & Kawato, M. (2024) · Neuroscience Research · doi:10.1016/j.neures.2024.01.007 · Cited by 8
“Early works have demonstrated that the basal ganglia, hosting a range of reinforcement learning processes, are connected to cortical areas through parallel and hierarchical loops (Draganski et al., 2008; Graybiel et al., 1994; Haruno and Kawato, 2006; Tanaka et al., 2004).”
BackgroundDiscounting of reward sequences: a test of competing formal models of hyperbolic discounting
Zarr, N., Alexander, W. H., & Brown, J. W. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00178 · Cited by 7
“While regions within the brain have been observed whose activity is consistent with distributed discount factors (Tanaka et al., 2004), it has not been established that such signals represent exponential rather than hyperbolic discount parameters.”
BackgroundNetwork Analysis of the Relationship Between Trait Depression and Impulsiveness Among Youth
Zhang, J., Li, K., Xue, Y., & Feng, Z. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.916332 · Cited by 6
“Depressive patients with anhedonia show reduced positive emotions for the future reward, which is explained as a lack of anticipatory pleasure (25).”
Background
Metadata
- Authors
- Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Japan Science and Technology Agency (JP)
- Hiroshima University (JP)
- Corresponding author
- Doya, Kenji
- Venue
- Nature Neuroscience
- Published
- 2004-07-04
- Volume
- 7
- Issue
- 8
- Pages
- 887-893
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2032610016
- Cited by
- 779
- References
- 54
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 9 · 2022: 29 · 2021: 28 · 2020: 32 · 2019: 25 · 2018: 32 · 2017: 37 · 2016: 40 · 2015: 45 · 2014: 50 · 2013: 42 · 2012: 47
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- reward prediction, time scales, cortico-basal ganglia loop, prediction error, insula, striatum, orbitofrontal cortex, dorsolateral prefrontal cortex, dorsal raphe nucleus, cerebellum, computational modeling, reinforcement learning
- MeSH terms
- Adult, Brain, Brain Mapping, Discrimination Learning, Female, Humans, Magnetic Resonance Imaging, Male, Markov Chains, Middle Aged, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.