Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Evaluation of both immediate and future outcomes of one's actions is a critical requirement for intelligent behavior. Using functional magnetic resonance imaging (fMRI), we investigated brain mechanisms for reward prediction at different time scales in a Markov decision task. When human subjects learned actions on the basis of immediate rewards, significant activity was seen in the lateral orbitofrontal cortex and the striatum. When subjects learned to act in order to obtain large future rewards while incurring small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects' performance data revealed graded maps of time scale within the insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior regions were involved in predicting future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In neurotypical individuals, the key brain loci that have been implicated in prediction are basal ganglia (68, 69), striatum (70, 71), anterior cingulate (72), and cerebellum (73, 74).”
From Autism as a disorder of prediction (2014) · cites this paper as [69]
Citations
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“, which has been successfully used for explaining reward-predictive activations of the midbrain dopaminergic system as well as those of the cortex and the striatu”
MethodEmotion, Decision Making and the Orbitofrontal Cortex
Bechara, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.295 · Cited by 3,589 · cited 1×
“Damage to the prefrontal cortex often impairs daily decision making, which requires assessment of future outcome”
BackgroundBarto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Basal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“The cortex and the basal ganglia both have parallel loop organization, with four major loops (limbic, cognitive, motor and oculomotor) and finer, topographic sub-loops within each major loo”
Background(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
The Orbitofrontal Cortex and Reward
Rolls, E. T. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.284 · Cited by 1,595
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001) · Neuron · doi:10.1016/s0896-6273(01)00303-8 · Cited by 1,404
Cited by
Items 401–500 of 798
İnce, E., & Üçok, A. (2017) · Clinical EEG and Neuroscience · doi:10.1177/1550059417746213 · Cited by 36
Aoki, N., Csillag, A., & Matsushima, T. (2006) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2006.05090.x · Cited by 43
“One possible explanation is that these areas coded the behavioral consequences of choices in distinct time domains, as has been suggested in recent human brain-imaging studies (McClure et al., 2004; Tanaka et al., 2004).”
BackgroundNeural substrates for sharing intention in action during face-to-face imitation
Miyata, K., Koike, T., Nakagawa, E., Harada, T., Sumiya, M., Yamamoto, T., & Sadato, N. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117916 · Cited by 35
Distinct cortico-striatal connections with subthalamic nucleus underlie facets of compulsivity
Morris, L. S., Baek, K., & Voon, V. (2016) · Cortex · doi:10.1016/j.cortex.2016.12.018 · Cited by 34
“However, the hippocampus has been shown to encode reward prediction (Tanaka et al., 2004), which is necessary for the reinforcement learning that drives model-free behaviour (Glascher, Daw, Dayan, & O'Doherty, 2010).”
Anggraini, D., Glasauer, S., & Wunderlich, K. (2018) · Scientific Reports · doi:10.1038/s41598-018-28241-z · Cited by 35
Decreased reward circuit connectivity during reward anticipation in major depression
Geugies, H., Groenewold, N. A., Meurs, M., Doornbos, B., de Klerk‐Sluis, J. M., van Eijndhoven, P., Roest, A. M., & Ruhé, H. G. (2022) · NeuroImage Clinical · doi:10.1016/j.nicl.2022.103226 · Cited by 37
“The insula has been linked to anticipating future rewards (Tanaka et al., 2004) and delayed gratification (Wittmann et al., 2007).”
BackgroundA Reinforcement Learning Model of Precommitment in Decision Making
Kurth-Nelson, Z. L., & Redish, A. D. (2010) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2010.00184 · Cited by 36
“During delay discounting tasks, there is a distribution across the striatum of areas correlated with different discount rates (Tanaka et al., 2004), consistent with the theory of a distributed set of agents exponentially discounting in parallel.”
Li, L., Cazzell, M. A., Zeng, L., & Liu, H. (2016) · Brain Imaging and Behavior · doi:10.1007/s11682-016-9580-z · Cited by 35
“It has been further suggested that males exhibit suppressed brain activation in the OFC, amygdala, and ventral striatum, with occasional losses in order to achieve longterm winnings (Tanaka et al. 2004; McCarrey et al. 2012; van den Bos et al. 2013).”
BackgroundChoosing Money over Drugs: The Neural Underpinnings of Difficult Choice in Chronic Cocaine Users
Wesley, M. J., Lohrenz, T. M., Koffarnus, M. N., McClure, S. M., Garza, R. D. L., Salas, R., Thompson-Lake, D. G. Y., Newton, T. F., Bickel, W. K., & Montague, P. R. (2014) · Journal of Addiction · doi:10.1155/2014/189853 · Cited by 32
Hadj‐Bouziane, F., Bénatru, I., Brovelli, A., Klinger, H., Thobois, S., Broussolle, E., Boussaoud, D., & Meunier, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00351 · Cited by 36
“…in humans: Deiber et al., 1997; Toni and Passingham, 1999; Toni et al., 2001a,b; Simon et al., 2002; Haruno et al., 2004; O’Doherty et al., 2004; Tanaka et al., 2004; Tricomi et al., 2004; Boettiger and D’Esposito, 2005; Delgado et al., 2005; Law et al., 2005; Grol et al., 2006; Haruno and…”
BackgroundMethodMaking decisions with a continuous mind
Scherbaum, S., Dshemuchadse, M., & Kalis, A. (2008) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.8.4.454 · Cited by 31
Neural Signals Related to Outcome Evaluation Are Stronger in CA1 than CA3
Lee, S.-H., Huh, N., Lee, J. W., Ghim, J.-W., Lee, I., & Jung, M. W. (2017) · Frontiers in Neural Circuits · doi:10.3389/fncir.2017.00040 · Cited by 31
“In this regard, significant value signals have been found in the human (Tanaka et al., 2004; Bornstein and Daw, 2013) and rat (Lee et al.”
BackgroundWittkuhn, L., Eppinger, B., Bartsch, L. M., Thurm, F., Korb, F. M., & Li, S. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.11.057 · Cited by 31
“…(BOLD) activity in the left dlPFC was associated with the behavior of taking actions associated with minor negative consequences (e.g., small monetary losses) in earlier states in order to gain a better net outcome (e.g., a larger reward) at a later state in the sequence (Tanaka et al., 2004).”
BackgroundMethodResultInfluentialSingle-cell memory trace imaging with immediate-early genes
Sauvage, M. M., Kitsukawa, T., & Atucha, E. (2019) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2019.108368 · Cited by 31
Soliva, J. C., Carmona, S., Fauquet, J., Hoekzema, E., Bulbena, A., Hilferty, J., & Vilarroya, O. (2009) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2009.04604.x · Cited by 25
Jessup, R. K., & O’Doherty, J. P. (2014) · European Journal of Neuroscience · doi:10.1111/ejn.12625 · Cited by 32
“Although we did replicate previous findings of a role for the medial prefrontal cortex in encoding expectations of future reward (Tanaka et al., 2004; Daw et al., 2006; Hampton et al., 2006), the study was not designed to focus on encoding of predictions about future outcomes, nor on encoding of…”
ResultPrevious cocaine self-administration disrupts reward expectancy encoding in ventral striatum
Burton, A. C., Bissonette, G. B., Vázquez, D., Blume, E. M., Donnelly, M., Heatley, K. C., Hinduja, A., & Roesch, M. R. (2018) · Neuropsychopharmacology · doi:10.1038/s41386-018-0058-0 · Cited by 29
Tanaka, S., O’Doherty, J. P., & Sakagami, M. (2019) · Nature Communications · doi:10.1038/s41467-019-11334-2 · Cited by 32
Self-citationNeural coding of computational factors affecting decision making
Dreher, J. (2013) · Progress in brain research · doi:10.1016/b978-0-444-62604-2.00016-2 · Cited by 29
Dopaminergic modulation of impulsive decision making in the rat insular cortex
Pattij, T., Schetters, D., & Schoffelmeer, A. N. M. (2014) · Behavioural Brain Research · doi:10.1016/j.bbr.2014.05.010 · Cited by 31
Neural basis of decision making guided by emotional outcomes
Katahira, K., Matsuda, Y.-T., Fujimura, T., Ueno, K., Asamizuya, T., Suzuki, C., Cheng, K., Okanoya, K., & Okada, M. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00564.2014 · Cited by 33
“The neural basis of decision making and associative learning based on reward or punishment has been intensively studied, and several brain regions, such as the striatum and the insula, are known to be involved in the learning processes (O’Doherty et al. 2003b, 2004; Seymour et al. 2004, 2005; Tanaka et al. 2004).”
Meade, C. S., Hobkirk, A. L., Towe, S. L., Chen, N., Bell, R. P., & Huettel, S. A. (2017) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2017.05.043 · Cited by 29
“The cognitive control network is increasingly recruited during “hard” choices in which the subjective value of the immediate and delayed rewards are equivalent compared to “easy” choices in which the values are highly discrepant (McClure et al., 2004; Tanaka et al., 2004).”
BackgroundNguyen, K. P., Fatt, C. C., Treacher, A., Mellema, C., Cooper, C. M., Jha, M. K., Kurian, B. T., Fava, M., McGrath, P. J., Weissman, M. M., et al. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2021.09.011 · Cited by 31
“Though traditionally associated with motor control, recent developments have strongly linked the cerebellum to reward processing and particularly error-based learning (33; 34).”
BackgroundChan, S. Y., Ong, Z. Y., Ngoh, Z. M., Chong, Y. S., Zhou, J. H., Fortier, M. V., Daniel, L. M., Qiu, A., Meaney, M. J., & Tan, A. P. (2022) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2022.101107 · Cited by 29
Empathy as a neuropsychological heuristic in social decision-making
Ramsøy, T. Z., Skov, M., Macoveanu, J., Siebner, H. R., & Fosgaard, T. R. (2014) · Social Neuroscience · doi:10.1080/17470919.2014.965341 · Cited by 34
“…in the valuation of response options (Koeneke, Pedroni, Dieckmann, Bosch, & Jäncke, 2008; Liu, Hairston, Schrier, & Fan, 2011; Piech et al., 2010; Tanaka et al., 2004), while dlPFC and the cingulate cortex are known to be involved in the evaluation between options and implementation of choices…”
BackgroundMeade, C. S., Cordero, D. M., Hobkirk, A. L., Metra, B. M., Chen, N., & Huettel, S. A. (2016) · Human Brain Mapping · doi:10.1002/hbm.23185 · Cited by 31
“Functional MRI (fMRI) research has implicated lateral and medial prefrontal cortices, ventral striatum, posterior cingulate, and insula in reward valuation and choice during temporal discounting [Kable and Glimcher, 2007; McClure et al., 2004; Tanaka et al., 2004].”
BackgroundNeurobiological processes during the Cambridge gambling task
Yazdi, K., Rumetshofer, T., Gnauer, M., Csillag, D., Rosenleitner, J., & Kleiser, R. (2018) · Behavioural Brain Research · doi:10.1016/j.bbr.2018.08.017 · Cited by 29
Anatomy and connectivity of the reward circuit
Haber, S. N. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374620-7.00001-7 · Cited by 33
MPTP-induced executive dysfunction is associated with altered prefrontal serotonergic function
Maiti, P., Gregg, L. C., & McDonald, M. P. (2015) · Behavioural Brain Research · doi:10.1016/j.bbr.2015.09.014 · Cited by 28
“Simultaneous activation of the dorsal raphe and dorsolateral PFC during delayed reinforcement has also been demonstrated in human subjects [66].”
Nakao, T., Kanayama, N., Katahira, K., Odani, M., Ito, Y., Hirata, Y., Nasuno, R., 繁, 尾., Hiramoto, R., Miyatani, M., et al. (2016) · Scientific Reports · doi:10.1038/srep32477 · Cited by 29
Ranking Regions, Edges and Classifying Tasks in Functional Brain Graphs by Sub-Graph Entropy
Sen, B., Chu, S.-H., & Parhi, K. K. (2019) · Scientific Reports · doi:10.1038/s41598-019-44103-8 · Cited by 32
Ageing differentially affects neural processing of different conflict types—an fMRI study
Korsch, M., Frühholz, S., & Herrmann, M. (2014) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2014.00057 · Cited by 28
“…Aron et al., 2003b) there is also evidence from neuroimaging studies including healthy participants demonstrating that the basal ganglia are also consistently found to be involved in tasks, which demand high cognitive control during decision making (Tanaka et al., 2004; Forstmann et al., 2008).”
ResultThe Expected Utility of Movement
Trommershäuser, J., Maloney, L. T., & Landy, M. S. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00008-7 · Cited by 32
“A number of studies suggest that reward value is encoded in the striatum and portions of prefrontal cortex (PFC) and orbitofrontal cortex (OFC) (Daw et al., 2006; Knutson et al., 2005; O’Doherty, 2004; Tanaka et al., 2005; Tom et al., 2007).”
BackgroundA Role for the Striatum in Regret-related Choice Repetition
Nicolle, A., Bach, D. R., Driver, J., & Dolan, R. J. (2010) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21510 · Cited by 28
“Tanaka et al. (2004, 2007) have found the dorsal striatum to be active when choosing larger delayed rewards in favor of smaller immediate rewards, supporting its role in the motivation of actions toward longer term goals, whereas more ventral regions of the striatum were active when choices were…”
BackgroundNeural Mechanism to Simulate a Scale-Invariant Future
Shankar, K. H., Singh, I., & Howard, M. W. (2016) · Neural Computation · doi:10.1162/neco_a_00891 · Cited by 30
Shimokawa, T., Suzuki, K., Misawa, T., & Miyagawa, K. (2009) · Neuroscience · doi:10.1016/j.neuroscience.2009.02.079 · Cited by 30
Separate Neural Networks for Gains and Losses in Intertemporal Choice
Zhang, Y., Xu, L., Liang, Z., Wang, K., Hou, B., Zhou, Y., Li, S., & Jiang, T. (2018) · Neuroscience Bulletin · doi:10.1007/s12264-018-0267-x · Cited by 28
Neuroscience, moral reasoning, and the law
Knabb, J. J., Welsh, R. K., Ziebell, J. G., & Reimer, K. S. (2009) · Behavioral Sciences & the Law · doi:10.1002/bsl.854 · Cited by 33
Onoda, K., Okamoto, Y., Kunisato, Y., Aoyama, S., Shishida, K., Okada, G., Tanaka, S., Schweighofer, N., Yamaguchi, S., Doya, K., et al. (2011) · Experimental Brain Research · doi:10.1007/s00221-011-2771-3 · Cited by 27
Self-citationInternal-Time Temporal Difference Model for Neural Value-Based Decision Making
Nakahara, H., & Kaveri, S. R. (2010) · Neural Computation · doi:10.1162/neco_a_00049 · Cited by 27
“…each subsystem contributed differentially to valuation at different timescales and behaved similar to the previously described multiple subsystems (Laibson, 1997; Tanaka et al., 2004; McClure et al., 2004; Montague et al., 2006; Fudenberg & Levine, 2006; Berns et al., 2007; McClure et al., 2007).”
BackgroundMethodInfluentialHow Costs Influence Decision Values for Mixed Outcomes
Talmi, D., & Pine, A. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00146 · Cited by 26
“…decision-making (for reviews in animal studies see Cardinal et al., 2004; Winstanley et al., 2006; Floresco et al., 2008a,b; and in humans Tanaka et al., 2004, 2007; McClure et al., 2004, 2007; Kable and Glimcher, 2007; Gregorios-Pippas et al., 2009; Luhmann et al., 2008; Ballard and…”
BackgroundOn the Nature, Modeling, and Neural Bases of Social Ties
van Winden, F. A. A. M., Stallen, M., & Ridderinkhof, K. R. (2008) · RePub (Erasmus University Rotterdam) · Cited by 27
Sex Differences in Time Perception During Smoking Abstinence
Ashare, R. L., & Kable, J. W. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntu260 · Cited by 24
Time to Abandon the Notion of Personal Choice in Dietary Counseling for Obesity?
Appelhans, B. M., Whited, M. C., Schneider, K. L., & Pagoto, S. L. (2011) · Journal of the American Dietetic Association · doi:10.1016/j.jada.2011.05.014 · Cited by 23
Yarosh, H. L., Hyatt, C. J., Meda, S. A., Jiantonio‐Kelly, R. E., Potenza, M. N., Assaf, M., & Pearlson, G. D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0088188 · Cited by 23
Hirter, K. N., Miller, E. N., Stimpson, C. D., Phillips, K. A., Hopkins, W. D., Hof, P. R., Sherwood, C. C., Lovejoy, C. O., & Raghanti, M. A. (2021) · Brain Structure and Function · doi:10.1007/s00429-021-02300-0 · Cited by 23
Time on timing: Dissociating premature responding from interval sensitivity in Parkinson's disease
Zhang, J., Nombela, C., Wolpe, N., Barker, R. A., & Rowe, J. B. (2016) · Movement Disorders · doi:10.1002/mds.26631 · Cited by 25
When theory and biology differ: The relationship between reward prediction errors and expectancy
Williams, C. C., Hassall, C. D., Trska, R., Holroyd, C. B., & Krigolson, O. E. (2017) · Biological Psychology · doi:10.1016/j.biopsycho.2017.09.007 · Cited by 24
“In humans, studies using both functional magnetic resonance imaging (Bray &O’Doherty, 2007; Brown & Braver, 2005; Haruno & Kawato, 2006; Jessup, Busemeyer, & Brown, 2010; Nieuwenhuis et al., 2005; Niv, Edlund, Dayan, & O’Doherty, 2012; O’Doherty et al., 2004; Roy et al., 2014; Silvetti, Seurinck, & Verguts, 2013; Tana…”
Wynn, S. C., Driessen, J. M. A., Glennon, J. C., Brazil, I. A., & Schutter, D. J. L. G. (2019) · The Cerebellum · doi:10.1007/s12311-019-01047-z · Cited by 24
Reinforcement Learning in a Nutshell
Heidrich-Meisner, V., Lauer, M., Igel, C., & Riedmiller, M. (2007) · Cited by 23
Functional Anatomy and Physiology of the Basal Ganglia: Non-motor Functions
Haber, S. N. (2008) · Humana Press eBooks · doi:10.1007/978-1-59745-360-8_2 · Cited by 25
The neural correlates of endowment effect without economic transaction
Votinov, M., Mima, T., Aso, T., Abe, M., Sawamoto, N., Shinozaki, J., & Fukuyama, H. (2010) · Neuroscience Research · doi:10.1016/j.neures.2010.05.006 · Cited by 26
“Studies using the reward-related tasks observed the orbitofrontal activities in response to reward-predicting signals (Tanaka et al. 2004; Tremblay and Schultz 1999), in processing a combined signal of expected value and risk (Tobler et al. 2009), and in decision-making during uncertainty (Huettel…”
Mechanisms of cerebellar contributions to cognition in humans
Bellebaum, C., Daum, I., & Suchan, B. (2012) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1161 · Cited by 23
Acute tryptophan depletion and sweet food consumption by overweight adults
Pagoto, S. L., Spring, B., McChargue, D., Hitsman, B., Smith, M., Appelhans, B., & Hedeker, D. (2009) · Eating Behaviors · doi:10.1016/j.eatbeh.2008.10.010 · Cited by 22
“…enhance preferences for immediate over delayed rewards (Hariri et al., 2006; Kelley, Schlitz, & Landry, 2005; McClure, Ericson, Laibson, Lowenstein, & Cohen, 2007; McClure, Laibson, Loewenstein, & Cohen, 2004; Tanaka et al., 2004), however other factors could have guided the first food consumed.”
Okada, K., Nakamura, K., & Kobayashi, Y. (2011) · Neural Plasticity · doi:10.1155/2011/579840 · Cited by 24
“The human DRN was activated when subjects learned to obtain large future rewards [119].”
BackgroundBerlijn, A. M., Huvermann, D. M., Schneider, S., Bellebaum, C., Timmann, D., Minnerop, M., & Peterburs, J. (2024) · The Cerebellum · doi:10.1007/s12311-024-01669-y · Cited by 25
“In addition, predictions of future compared to immediate rewards were again associated with activation in the left lateral cerebellum, revealing that the time scale of the reward had an influence on how the cerebellum generated the prediction [109].”
BackgroundMethodResultCoding of the long-term value of multiple future rewards in the primate striatum
Yamada, H., Inokawa, H., Matsumoto, N., Ueda, Y., Enomoto, K., & Kimura, M. (2012) · Journal of Neurophysiology · doi:10.1152/jn.00289.2012 · Cited by 23
“This is one of the convincing methods that have been used to find the representation of long-term values with heterogeneous discount rates (Tanaka et al. 2004), although it is a different approach to find the neural coding of subjective values of rewards estimated from the history of behavioral…”
BackgroundMethodInfluentialTemporal chunking as a mechanism for unsupervised learning of task-sets
Bouchacourt, F., Palminteri, S., Koechlin, É., & Ostojic, S. (2020) · eLife · doi:10.7554/elife.50469 · Cited by 25
Effort, success, and side of lesion determine arm choice in individuals with chronic stroke
Kim, S., Han, C. E., Kim, B., Winstein, C. J., & Schweighofer, N. (2021) · Journal of Neurophysiology · doi:10.1152/jn.00532.2020 · Cited by 25
Dissociation of BOLD responses to reward prediction errors and reward receipt by a model comparison
Rohe, T., Weber, B., & Fliessbach, K. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08125.x · Cited by 22
“…studies reporting activation effects of RPE should compare this effect to a simpler effect of RR, which is usually not done in studies using an RPE regressor extracted from computational temporal-difference models (e.g. O’Doherty et al., 2003; Tanaka et al., 2004; Pessiglione et al., 2006).”
Electrophysiological markers of the motivational salience of delay imposition and escape
Broyd, S. J., Richards, H. J., Helps, S. K., Chronaki, G., Bamford, S., & Sonuga-Barke, E. J. S. (2012) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2012.02.003 · Cited by 25
“A number of models have been proposed to explain the disproportionate power of immediate rewards and their influence on choice (Ainslie, 1975; Marco, et al., 2009; Sagvolden, et al., 2005; Sonuga-Barke, 2002; Sonuga- Barke, et al., 2010; Tripp & Wickens, 2008).”
Dual-System Learning Models and Drugs of Abuse
Simon, D. A., & Daw, N. D. (2011) · Springer eBooks · doi:10.1007/978-1-4614-0751-5_5 · Cited by 24
“…values in other areas of the brain, including prefrontal cortex, striatum, and parietal regions (Delgado et al. 2000; Arkadir et al. 2004; Tanaka et al. 2004; Samejima et al. 2005; Plassmann et al. 2007; Tom et al. 2007; Kable and Glimcher 2007; Hare et al. 2008; Kim et al. 2009;…”
BackgroundHan, D., Doya, K., & Tani, J. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.06.002 · Cited by 23
“However, (Tanaka et al., 2016; Enomoto et al., 2011) have shown that dopamine neurons in mammalian brains encode value functions with different region-specific discount factors.”
BackgroundResultSelf-citationWorking‐memory training: Effects on delay discounting in male long evans rats
Renda, C. R., Stein, J. S., & Madden, G. J. (2014) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.115 · Cited by 23
“Studies supporting the CNDS theory include those demonstrating that a) separate neural systems are activated when choosing SSRs versus LLRs (McClure et al., 2007; McClure et al., 2004; Tanaka et al., 2004), b) transcranial magnetic stimulation of brain structures responsible for executive-system behavior affects delay…”
BackgroundGentili, R. J., Papaxanthis, C., Ebadzadeh, M. M., Eskiizmirliler, S., Ouanezar, S., & Darlot, C. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005176 · Cited by 23
“Also, highly neural recurrent structures such as the hippocampus [56,57], the prefrontal [58,59] and the orbitofrontal [60,61] cortex seem to perform associative prediction that would reflect outcome expectancies, providing thus, an internal representation of the consequences likely to follow a…”
BackgroundTops, M., & Boksem, M. A. S. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00021 · Cited by 22
“…of time scale within the right IFG–insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior striatal regions (and dorsolateral prefrontal cortex, posterior cingulate cortex) were involved in predicting future rewards (Tanaka et al., 2004).”
BackgroundDifferential magnitude coding of gains and omitted rewards in the ventral striatum
Pedroni, A. F., Koeneke, S., Velickaite, A., & Jäncke, L. (2011) · Brain Research · doi:10.1016/j.brainres.2011.07.019 · Cited by 22
“Congruently, in some neuroimaging studies the BOLDsignal in the ventral striatum correlates with reward magnitude (McClure et al., 2004, Tanaka et al., 2004) and not reward magnitude dependent prediction errors.”
BackgroundResultThe neural predictors of choice preference in intertemporal choice
Liu, L., & Feng, T. (2011) · Brain Research · doi:10.1016/j.brainres.2011.12.018 · Cited by 25
Multiple model-based reinforcement learning explains dopamine neuronal activity
Bertin, M., Schweighofer, N., & Doya, K. (2007) · Neural Networks · doi:10.1016/j.neunet.2007.04.028 · Cited by 21
“As serotonin appears to control the selection of different corticobasal ganglia loops in reward prediction at different time scales (Schweighofer, Tanaka, et al., 2004; Tanaka et al., 2004), serotonergic projections could carry the responsibility signals necessary for learning and controlling appropriate reward predic…”
BackgroundInfluentialSelf-citationComputational algorithms and neuronal network models underlying decision processes
Sakai, Y., Okamoto, H., & Fukai, T. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.034 · Cited by 23
“, 1997), and the mechanisms to generate such neuronal responses have been suggested by modeling studies (Dayan & Balleine, 2002; Doya, 2000; Haruno et al., 2004; Houk, Davis, & Beiser, 1994; O’Doherty et al., 2004; Schultz, 2004; Seung, 2003; Tanaka et al., 2004; Wang, 2002).”
BackgroundInfluentialBaranowska‐Bosiacka, I., Listos, J., Gutowska, I., Machoy‐Mokrzyńska, A., Kolasa, A., Tarnowski, M., Puchałowicz, K., Prokopowicz, A., Talarek, S., Listos, P., et al. (2015) · Toxicology · doi:10.1016/j.tox.2015.10.003 · Cited by 22
“It is strongly involved in various aspects of the state of dependence (Delgato et al., 2003, Everitt et al., 2005, Tanaka et al., 2004).”
BackgroundIvanova, E., Panayotova, T., Grechenliev, I., Peshev, B., Kolchakova, P. Y., & Milanova, V. K. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.795344 · Cited by 25
Dexter, T. D., Roberts, B. Z., Ayoub, S. M., Noback, M. N., Barnes, S. A., & Young, J. W. (2024) · Journal of Neurochemistry · doi:10.1111/jnc.16243 · Cited by 22
“…expected value of reward outcome and during reward receipt across learning (Daw et al., 2006; Gläscher et al., 2009 ; Liu, Hairston, et al., 2011; Tanaka et al., 2004), together suggesting that while the vmPFC appeared to monitor and update reward value, its contributions to learning are…”
Her, E. S., Huh, N., Kim, J., & Jung, M. W. (2016) · Scientific Reports · doi:10.1038/srep27056 · Cited by 21
Uncertainty-Dependent Extinction of Fear Memory in an Amygdala-mPFC Neural Circuit Model
Li, Y., Nakae, K., Ishii, S., & Naoki, H. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005099 · Cited by 23
van de Ven, V. G., Lee, C., Lifanov, J., Kochs, S., Jansma, H., & Weerd, P. D. (2020) · Hippocampus · doi:10.1002/hipo.23205 · Cited by 22
“timed stimulus-response gaps (Buhusi & Meck, 2005; Coull et al., 2004; Rao et al., 2001; Tanaka et al., 2004; van Rijn et al., 2014; Wiener, Turkeltaub, & Coslett, 2010), but also extends it to processing”
BackgroundResultSchommartz, I., Dix, A., Passow, S., & Li, S. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.605190 · Cited by 22
“Previous research has investigated the underlying neural correlates of individuals’ sequential decision-making performance by using different variants of a deterministic three-stage Markov task (Tanaka et al., 2004; Eppinger et al., 2015; Wittkuhn et al., 2018).”
BackgroundResultInfluentialPrins, N. W., Sanchez, J. C., & Prasad, A. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00111 · Cited by 21
Ikegami, A., Olsen, C. M., D’Souza, M. S., & Duvauchelle, C. L. (2007) · Behavioral Neuroscience · doi:10.1037/0735-7044.121.2.389 · Cited by 21
“These results resemble human fMRI findings showing dissociations between subcortical and cortical functioning during reward and nonreward anticipation, outcome, and prediction errors (Knutson & Cooper, 2005; Knutson, Fong, Adams, Varner, & Hommer, 2001), and parallel clinical work revealing preferential caudate/ventra…”
BackgroundResultTaniguchi, T., & Sawaragi, T. (2007) · Advanced Robotics · doi:10.1163/156855307781389374 · Cited by 20
“Basal ganglia is charged with reinforcement learning [8, 9].”
BackgroundAsahi, T., Uwano, T., Eifuku, S., Tamura, R., Endo, S., Ono, T., & Nishijo, H. (2006) · Neuroscience · doi:10.1016/j.neuroscience.2006.08.008 · Cited by 19
Five Breakthroughs: A First Approximation of Brain Evolution From Early Bilaterians to Humans
Bennett, M. S. (2021) · Frontiers in Neuroanatomy · doi:10.3389/fnana.2021.693346 · Cited by 20
Sakaki, M., Meliß, S., Murayama, K., Yomogida, Y., Matsumori, K., Sugiura, A., Matsumoto, M., & Matsumoto, K. (2022) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-022-01046-4 · Cited by 21
Neural Representation of a Target Auditory Memory in a Cortico-Basal Ganglia Pathway
Achiro, J. M., & Bottjer, S. W. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.0710-13.2013 · Cited by 22
“Parallel cortico-basal ganglia loops in mammals process distinct but related aspects of sequential motor and reward-based learning (Hikosaka et al., 1999; O’Doherty et al., 2004; Tanaka et al., 2004; Yin et al., 2009; Thorn et al., 2010).”
BackgroundNeural and Genetic Correlates of the Social Sharing of Happiness
Matsunaga, M., Kawamichi, H., Umemura, T., Hori, R., Shibata, E., Kobayashi, F., Suzuki, K., Ishii, K., Ohtsubo, Y., Noguchi, Y., et al. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00718 · Cited by 22
“…between happiness and the mentalizing/theory-of-mind system is conceivable because the mPFC plays a pivotal role in future reward estimation (Tanaka et al., 2004; Rushworth et al., 2011), which may be a key factor in enhancing subjective happiness (Sheldon and Houser-Marko, 2001;…”
BackgroundThams, F., Kuzmina, A. V., Backhaus, M., Li, S., Grittner, U., Antonenko, D., & Flöel, A. (2020) · Alzheimer s Research & Therapy · doi:10.1186/s13195-020-00692-5 · Cited by 20
Robertazzi, F., Vissani, M., Schillaci, G., & Falotico, E. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.06.020 · Cited by 20
Williams, J. (2008) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2008.01921.x · Cited by 18
Dynamics of human serotonin synthesis differentially link to reward anticipation and feedback
Hahn, A., Reed, M. B., Murgaš, M., Vraka, C., Klug, S., Schmidt, C., Godbersen, G. M., Eggerstorfer, B., Gomola, D., Silberbauer, L., et al. (2024) · Molecular Psychiatry · doi:10.1038/s41380-024-02696-1 · Cited by 20
Liu, X., Liu, S., Huang, R., Chen, X., Xie, Y., Ma, R., Luo, Y., Bu, J., & Zhang, X. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00461 · Cited by 19
Katahira, K., & Toyama, A. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008738 · Cited by 19
Leone, C. M., Galosi, S., Mollica, C., Fortunato, M., Possidente, C., Milone, V., Misuraca, S., Berillo, L., Truini, A., Cruccu, G., et al. (2021) · European Journal of Pain · doi:10.1002/ejp.1793 · Cited by 18
“Indeed, previous functional neuroimaging and postmortem studies in patients with suicidal behaviour (see van Heeringen & Mann, 2014 for a comprehensive review) disclosed structural and functional changes of the anterior cingulate and the insula associated with suicide (van Heeringen & Mann, 2014; Tanaka et al. 2004; G…”
BackgroundLa notion d'addiction en économie : La théorie du choix rationnel à l'épreuve
Massin, S. (2012) · Revue d économie politique · doi:10.3917/redp.215.0713 · Cited by 22
Understanding decision neuroscience: A multidisciplinary perspective and neural substrates
Miyapuram, K. P., & Pammi, V. S. C. (2013) · Progress in brain research · doi:10.1016/b978-0-444-62604-2.00014-9 · Cited by 19
Neural Correlates of Effective Learning in Experienced Medical Decision-Makers
Downar, J. H., Bhatt, M. A., & Montague, P. R. (2011) · PLoS ONE · doi:10.1371/journal.pone.0027768 · Cited by 22
Flack, K. D., Anderson, R. E., McFee, K. F., Kryscio, R., & Rush, C. R. (2022) · Physiology & Behavior · doi:10.1016/j.physbeh.2022.113711 · Cited by 17
Ez‐zizi, A., Farrell, S., Leslie, D. S., Malhotra, G., & Ludwig, C. J. H. (2023) · Computational Brain & Behavior · doi:10.1007/s42113-022-00165-y · Cited by 21
“Even though RL has been shown to be a viable proxy for human reward-based learning under expected reward uncertainty (e.g. Paulus et al., 2004; Tanaka et al., 2004), a recent body of work has found support for learning by sampling mechanism (Bornstein et al., 2017; Bornstein & Norman, 2017; Hochman…”
BackgroundStimulus-Dependent Adjustment of Reward Prediction Error in the Midbrain
Takemura, H., Samejima, K., Vogels, R., Sakagami, M., & Okuda, J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028337 · Cited by 18
Delay discounting and affective priming in individuals with negative schizotypy
Cai, X., Weigl, M., Liu, B.-H., Cheung, E. F. C., Ding, J., & Chan, R. C. K. (2018) · Schizophrenia Research · doi:10.1016/j.schres.2018.12.040 · Cited by 17
Self-Controlled Choice Arises from Dynamic Prefrontal Signals That Enable Future Anticipation
Tanaka, D., Aoki, R., Suzuki, S., Takeda, M., Nakahara, K., & Jimura, K. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.1702-20.2020 · Cited by 20
“…that valuation of the reward was unchanged throughout the entire experiment, and that participants made choices in the same way as they make in the real world (McClure et al., 2004, 2007; Tanaka et al., 2004; Hariri et al., 2006; Kable and Glimcher, 2007; Shamosh et al., 2008; Hare et al., 2009).”
BackgroundInfluential
Metadata
- Authors
- Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Japan Science and Technology Agency (JP)
- Hiroshima University (JP)
- Corresponding author
- Doya, Kenji
- Venue
- Nature Neuroscience
- Published
- 2004-07-04
- Volume
- 7
- Issue
- 8
- Pages
- 887-893
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2032610016
- Cited by
- 779
- References
- 54
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 9 · 2022: 29 · 2021: 28 · 2020: 32 · 2019: 25 · 2018: 32 · 2017: 37 · 2016: 40 · 2015: 45 · 2014: 50 · 2013: 42 · 2012: 47
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- reward prediction, time scales, cortico-basal ganglia loop, prediction error, insula, striatum, orbitofrontal cortex, dorsolateral prefrontal cortex, dorsal raphe nucleus, cerebellum, computational modeling, reinforcement learning
- MeSH terms
- Adult, Brain, Brain Mapping, Discrimination Learning, Female, Humans, Magnetic Resonance Imaging, Male, Markov Chains, Middle Aged, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.