Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Evaluation of both immediate and future outcomes of one's actions is a critical requirement for intelligent behavior. Using functional magnetic resonance imaging (fMRI), we investigated brain mechanisms for reward prediction at different time scales in a Markov decision task. When human subjects learned actions on the basis of immediate rewards, significant activity was seen in the lateral orbitofrontal cortex and the striatum. When subjects learned to act in order to obtain large future rewards while incurring small immediate losses, the dorsolateral prefrontal cortex, inferior parietal cortex, dorsal raphe nucleus and cerebellum were also activated. Computational model-based regression analysis using the predicted future rewards and prediction errors estimated from subjects' performance data revealed graded maps of time scale within the insula and the striatum: ventroanterior regions were involved in predicting immediate rewards and dorsoposterior regions were involved in predicting future rewards. These results suggest differential involvement of the cortico-basal ganglia loops in reward prediction at different time scales.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In neurotypical individuals, the key brain loci that have been implicated in prediction are basal ganglia (68, 69), striatum (70, 71), anterior cingulate (72), and cerebellum (73, 74).”
From Autism as a disorder of prediction (2014) · cites this paper as [69]
Citations
Cites
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“, which has been successfully used for explaining reward-predictive activations of the midbrain dopaminergic system as well as those of the cortex and the striatu”
MethodEmotion, Decision Making and the Orbitofrontal Cortex
Bechara, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.295 · Cited by 3,589 · cited 1×
“Damage to the prefrontal cortex often impairs daily decision making, which requires assessment of future outcome”
BackgroundBarto, A. G. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,976
Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
Basal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“The cortex and the basal ganglia both have parallel loop organization, with four major loops (limbic, cognitive, motor and oculomotor) and finer, topographic sub-loops within each major loo”
Background(2020) · Machine Learning and Visual Perception · doi:10.1515/9783110595567-011 · Cited by 1,930
The Orbitofrontal Cortex and Reward
Rolls, E. T. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.284 · Cited by 1,595
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001) · Neuron · doi:10.1016/s0896-6273(01)00303-8 · Cited by 1,404
Cited by
The reward circuit: linking primate anatomy and human imaging
Haber, S. N., & Knutson, B. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.129 · Cited by 3,718
“Thus, sequentially over time, this suggests a mechanism through which activity occurs across reward-processing episodes during the course of learning (Tanaka et al , 2004).”
BackgroundSeparate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
A framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
The evolution of foresight: What is mental time travel, and is it unique to humans?
Suddendorf, T., & Corballis, M. C. (2007) · Behavioral and Brain Sciences · doi:10.1017/s0140525x07001975 · Cited by 2,031
“In one functional magnetic resonance imaging (fMRI) study, for example, brain activity was recorded while human subjects learned actions in a Markov decision task that would bring either immediate or future rewards (Tanaka et al. 2004).”
Bartra, O., McGuire, J. T., & Kable, J. W. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.02.063 · Cited by 1,927
“Both these regions are considered integral in representing SV, and their internal organization is a topic of growing interest (e.g., Seymour et al., 2007; Smith et al., 2010; Tanaka et al., 2004; Voorn et al., 2004).”
Inhibition and impulsivity: behavioral and neural basis of response control
Bari, A., & Robbins, T. W. (2013) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2013.06.005 · Cited by 1,921
The neural correlates of subjective value during intertemporal choice
Kable, J. W., & Glimcher, P. W. (2007) · Nature Neuroscience · doi:10.1038/nn2007 · Cited by 1,916
Social cognition and the brain: A meta-analysis
Van Overwalle, F. (2009) · Human Brain Mapping · doi:10.1002/hbm.20547 · Cited by 1,792
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Habits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584
Social reward requires coordinated activity of nucleus accumbens oxytocin and serotonin
Dölen, G., Darvishzadeh, A., Huang, K. W., & Malenka, R. C. (2013) · Nature · doi:10.1038/nature12518 · Cited by 1,161
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
Time Discounting for Primary Rewards
McClure, S. M., Ericson, K. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.4246-06.2007 · Cited by 971
The neural basis of human moral cognition
Moll, J., Zahn, R., de Oliveira-Souza, R., Krueger, F., & Grafman, J. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1768 · Cited by 930
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925
Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2010.12.012 · Cited by 916
Human fronto–mesolimbic networks guide decisions about charitable donation
Moll, J., Krueger, F., Zahn, R., Pardini, M., de Oliveira-Souza, R., & Grafman, J. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0604475103 · Cited by 900
Peters, J., & Büchel, C. (2010) · Neuron · doi:10.1016/j.neuron.2010.03.026 · Cited by 871
“DLPFC and ACC support various forms of decision-making (Rushworth and Behrens, 2008; Tanaka et al., 2004) including intertemporal choice (Figner et al., 2010; Kim et al., 2008; McClure et al., 2004, 2007).”
BackgroundChoice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
The orbitofrontal cortex and beyond: From affect to decision-making
Rolls, E. T., & Grabenhorst, F. (2008) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2008.09.001 · Cited by 807
“Further evidence in a variety of tasks implicates a related and partly overlapping region of the ventromedial prefrontal cortex with expected value (Daw et al., 2006; Hare et al., 2008; Kim et al., 2006; Tanaka et al., 2004).”
BackgroundBeckmann, M., Johansen-Berg, H., & Rushworth, M. F. S. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.3328-08.2009 · Cited by 789
“…in the last 3 years that focused on reward expectation and often using computationally derived regressors to estimate subjects’ reward expectations (Tanaka et al., 2004; Knutson and Cooper, 2005; Reuter et al., 2005; Daw and Doya, 2006; Hampton et al., 2006; Kim et al., 2006; Rolls et al., 2008)…”
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769
“If the region encompassed multiple anatomical regions with separate significance peaks, the threshold was raised until the contrast values for the region of interest could be plotted separately.”
BackgroundCerebellar networks with the cerebral cortex and basal ganglia
Bostan, A. C., Dum, R. P., & Strick, P. L. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.03.003 · Cited by 759
Functional magnetic resonance imaging of reward prediction
Knutson, B., & Cooper, J. C. (2005) · Current Opinion in Neurology · doi:10.1097/01.wco.0000173463.24758.f6 · Cited by 736
Informatic parcellation of the network involved in the computation of subjective value
Clithero, J. A., & Rangel, A. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nst106 · Cited by 706
Haber, S. N., Kim, K.-S., Mailly, P., & Calzavara, R. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.0271-06.2006 · Cited by 686
“Nonetheless, rewardresponsive activation is not restricted to the ventral striatum but is found throughout the striatum (Hassani et al., 2001; Takikawa et al., 2002; Tanaka et al., 2004).”
BackgroundResultThe basal ganglia: learning new tricks and loving it
Graybiel, A. M. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.10.006 · Cited by 649
Brain activities associated with gaming urge of online gaming addiction
Ko, C.-H., Liu, G.-C., Hsiao, S., Yen, J.-Y., Yang, M.-J., Lin, W.-C., Yen, C.-F., & Chen, C.-S. (2009) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2008.09.012 · Cited by 666
Boorman, E. D., Behrens, T. E. J., Woolrich, M. W., & Rushworth, M. F. S. (2009) · Neuron · doi:10.1016/j.neuron.2009.05.014 · Cited by 665
“This perspective accords well with recent demonstrations that vmPFC and other prefrontal regions are sensitive to both potential gains and costs during decision making (Hare et al., 2008; Kable and Glimcher, 2007; Kim et al., 2008; Rushworth et al., 2007; Tanaka et al., 2004; Tom et al., 2007).”
Decision making, impulsivity and time perception
Wittmann, M., & Paulus, M. P. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.10.004 · Cited by 602
Doya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642
“In functional brain imaging studies using a game in a dynamic environment, the dorsolateral prefrontal cortex, dorsal premotor cortex, parietal cortex and insula are more activated in conditions requiring long-term prediction of rewards rather than in conditions requiring short-term prediction”
BackgroundSelf-citationFractionating impulsivity: neuropsychiatric implications
Dalley, J. W., & Robbins, T. W. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2017.8 · Cited by 607
Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories
Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z., & Graybiel, A. M. (2005) · Nature · doi:10.1038/nature04053 · Cited by 595
Separate neural pathways process different decision costs
Rudebeck, P. H., Walton, M. E., Smyth, A. N., Bannerman, D. M., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1756 · Cited by 580
Postuma, R. B., & Dagher, A. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhj088 · Cited by 579
Autism as a disorder of prediction
Sinha, P., Kjelgaard, M. M., Gandhi, T. K., Tsourides, K., Cardinaux, A. L., Pantazis, D., Diamond, S. P., & Held, R. M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1416797111 · Cited by 563
Effort-Based Cost–Benefit Valuation and the Human Brain
Croxson, P. L., Walton, M. E., O'Reilly, J. X., Behrens, T. E. J., & Rushworth, M. F. S. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4515-08.2009 · Cited by 560
“Previous fMRI studies have also reported insula activity related to reward expectation, plus reward expectation uncertainty (Elliott et al., 2000; Tanaka et al., 2004; Knutson et al., 2005; Preuschoff et al., 2008).”
BackgroundEarly Adverse Experiences and the Developing Brain
Bick, J., & Nelson, C. A. (2016) · Neuropsychopharmacology · doi:10.1038/npp.2015.252 · Cited by 559
“Some portions support cognitive control and regulation, while others support affective regulation associated with motivation and reward sensitivity (Schultz et al, 2000; Tanaka et al, 2004).”
BackgroundAddiction as a Computational Process Gone Awry
Redish, A. D. (2004) · Science · doi:10.1126/science.1102384 · Cited by 540
Iacono, W. G., Malone, S. M., & McGue, M. (2008) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.4.022007.141157 · Cited by 554
van Heeringen, K., & Mann, J. J. (2014) · The Lancet Psychiatry · doi:10.1016/s2215-0366(14)70220-2 · Cited by 550
The neurobiology and genetics of impulse control disorders: Relationships to drug addictions
Brewer, J. A., & Potenza, M. N. (2008) · Biochemical Pharmacology · doi:10.1016/j.bcp.2007.06.043 · Cited by 511
Determining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529
“…by significant responses in orbitofrontal cortex related to subsequent behavioral choice during reversal learning (O’Doherty et al., 2003), and reports of expected value signals in these areas during performance of instrumental choice tasks (Tanaka et al., 2004; Daw et al., 2006; Kim et al., 2006).”
BackgroundModel‐Based fMRI and Its Application to Reward Learning and Decision Making
O'DOHERTY, J. P., HAMPTON, A., & KIM, H. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.022 · Cited by 508
A unified framework for addiction: Vulnerabilities in the decision process
Redish, A. D., Jensen, S., & Johnson, A. (2008) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0800472x · Cited by 501
Learning from Sensory and Reward Prediction Errors during Motor Adaptation
Izawa, J., & Shadmehr, R. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002012 · Cited by 477
Separate Valuation Subsystems for Delay and Effort Decision Costs
Prévost, C., Pessiglione, M., Météreau, E., Cléry-Melin, M.-L., & Dreher, J.-C. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.2752-10.2010 · Cited by 499
“…condition, the functional clusters of interest were the ventromedial prefrontal cortex (vmPFC) and the ventral striatum because of their reported role in coding the subjective value of delayed rewards in the monetary domain (Tanaka et al., 2004; Kable and Glimcher, 2007; Peters and Büchel, 2009).”
BackgroundHampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
“However, it is possible that behavior may be controlled by multiple parallel learning systems, each with distinct model parameters (Tanaka et al., 2004; Glascher et al., 2005), and as such these multiple learning systems would not be discriminated using our approach.”
BackgroundThe computational neurobiology of learning and reward
Daw, N. D., & Doya, K. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.006 · Cited by 465
Self-citationDifferential Encoding of Losses and Gains in the Human Striatum
Seymour, B., Daw, N., Dayan, P., Singer, T., & Dolan, R. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0400-07.2007 · Cited by 462
“…(BOLD) signal in the human striatum measured using functional magnetic resonance imaging (fMRI) covaries closely with key learning signals used by abstract learning models (O’Doherty et al., 2003; Haruno et al., 2004; Seymour et al., 2004; Tanaka et al., 2004, 2006; Yacubian et al., 2006).”
MethodKim, H., Shimojo, S., & O'Doherty, J. P. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040233 · Cited by 453
Collins, A. G. E., & Frank, M. J. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07980.x · Cited by 417
“…data, including electrophysiology (Corrado & Doya, 2007), functional magnetic resonance imaging (O’Doherty et al. , 2004, 2007; Seymour et al. , 2004; Tanaka et al. , 2004, 2006; Daw & Doya, 2006; Pessiglione et al. , 2006; Scho¨nberg et al. , 2007; Brovelli et al. , 2008; Kahnt et al. , 2009),…”
Opponency revisited: competition and cooperation between dopamine and serotonin
Boureau, Y.-L., & Dayan, P. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.151 · Cited by 456
Influence of Reward Delays on Responses of Dopamine Neurons
Kobayashi, S., & Schultz, W. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1600-08.2008 · Cited by 430
“The human striatum and orbitofrontal cortex (OFC) showed greater hemodynamic response to immediate rather than delayed reward (McClure et al., 2004; Tanaka et al., 2004).”
BackgroundOpponent appetitive-aversive neural processes underlie predictive learning of pain relief
Seymour, B., O'Doherty, J. P., Koltzenburg, M., Wiech, K., Frackowiak, R., Friston, K., & Dolan, R. (2005) · Nature Neuroscience · doi:10.1038/nn1527 · Cited by 439
A Scalable Population Code for Time in the Striatum
Mello, G. B. M., Soares, S., & Paton, J. J. (2015) · Current Biology · doi:10.1016/j.cub.2015.02.036 · Cited by 392
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Serotonin in Affective Control
Dayan, P., & Huys, Q. J. M. (2009) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135607 · Cited by 381
“…factor (interest rate) that allows distant rewards and punish-ments to be weighted against proximal ones, and they used fMRI data to link this hypothesis to changes in the topographic structure of the representation of predictions and prediction errors across the striatum (Tanaka et al. 2004).”
Basal ganglia orient eyes to reward
Hikosaka, O., Nakamura, K., & Nakahara, H. (2006) · Journal of Neurophysiology · doi:10.1152/jn.00458.2005 · Cited by 377
“This may suggest that reward-predicting mechanisms are widely distributed in the brain (O'Doherty et al. 2004) and are differentially involved in reward prediction at different reward delays (Mc- Clure et al. 2004a; Tanaka et al. 2004).”
BackgroundLaureiro‐Martínez, D., Brusoni, S., Canessa, N., & Zollo, M. (2015) · Strategic Management Journal · doi:10.1002/smj.2221 · Cited by 346
“…in exploitation choices (Daw et al., 2006) and in coding of the expected value (or other variables) related to the chosen option (Behrens et al., 2008; Daw et al., 2006; Hampton, Bossaerts, and O'Doherty, 2006; Hare et al., 2008; Kable and Glimcher, 2007; Knutson et al., 2005; Tanaka et al., 2004).”
BackgroundThe roots of modern justice: cognitive and neural foundations of social norms and their enforcement
Buckholtz, J. W., & Marois, R. (2012) · Nature Neuroscience · doi:10.1038/nn.3087 · Cited by 286
The place of dopamine in the cortico-basal ganglia circuit
Haber, S. N. (2014) · Neuroscience · doi:10.1016/j.neuroscience.2014.10.008 · Cited by 364
A critical review of sex differences in decision-making tasks: Focus on the Iowa Gambling Task
van den Bos, R., Homberg, J., & de Visser, L. (2013) · Behavioural Brain Research · doi:10.1016/j.bbr.2012.10.002 · Cited by 325
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346
“TDRL models have recently found additional support from fMRI experiments examining changes in value under careful economic controls (McClure et al., 2003; Tanaka et al., 2004; Paulus et al., 2004; O’Doherty, 2004).”
MethodThe orbitofrontal cortex and emotion in health and disease, including depression
Rolls, E. T. (2019) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.09.021 · Cited by 329
“Further evidence in a variety of tasks implicates a related and partly overlapping region of the ventromedial prefrontal cortex with expected value (Daw et al., 2006; Hare et al., 2008; Kim et al., 2006; Tanaka et al., 2004).”
Noreika, V., Falter, C. M., & Rubia, K. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2012.09.036 · Cited by 339
“…2009; Christakou et al., 2011; Hinvest et al., 2011; Tanaka et al., 2004, 2007; Wittmann et al., 2010), the dorsal striatum (Hinvest et al., 2011; Tanaka et al., 2004, 2007), and the cerebellum (Bickel et al., 2009; Christakou et al., 2011; Tanaka et al., 2004, 2007; Wittmann et al., 2010),…”
InfluentialPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
“(However, correlates of prediction errors for γ = 0 have been observed using human fMRI [46]).”
BackgroundMethodKable, J. W., & Glimcher, P. W. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00177.2009 · Cited by 316
Younger, J., Aron, A., Parke, S., Chatterjee, N., & Mackey, S. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013309 · Cited by 314
“The corticostriatal reward network provides one route by which strongly valenced external cues may reduce the experience of pain. Cue-evoked reward prediction is known to involve orbitofrontal and dorsolateral prefrontal cortex activity in humans”
BackgroundReward Improves Long-Term Retention of a Motor Memory through Induction of Offline Memory Gains
Abe, M., Schambra, H., Wassermann, E. M., Luckenbaugh, D., Schweighofer, N., & Cohen, L. G. (2011) · Current Biology · doi:10.1016/j.cub.2011.02.030 · Cited by 313
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Dopamine, Time, and Impulsivity in Humans
Pine, A., Shiner, T., Seymour, B., & Dolan, R. J. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.6028-09.2010 · Cited by 300
Appelhans, B. M., Woolf, K., Pagoto, S. L., Schneider, K. L., Whited, M. C., & Liebman, R. (2011) · Obesity · doi:10.1038/oby.2011.57 · Cited by 278
Seger, C. A. (2008) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2007.07.010 · Cited by 274
“Sensitivity increases to delayed rewards along a gradient progressing dorsally and posteriorly into head of the caudate nucleus (Tanaka et al., 2004).”
BackgroundThe Cognitive Neuroscience of Human Decision Making: A Review and Conceptual Framework
Fellows, L. K. (2004) · Behavioral and Cognitive Neuroscience Reviews · doi:10.1177/1534582304273251 · Cited by 265
“The immediate reward condition was related to activation in the OFC, whereas the trade-off choice also recruited DLF, among other areas (Tanaka et al., 2004).”
BackgroundValidation of Decision-Making Models and Analysis of Decision Variables in the Rat Basal Ganglia
Ito, M., & Doya, K. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.6157-08.2009 · Cited by 286
“The firing activity of midbrain dopamine neurons (Schultz et al., 1997) and the blood oxygen level-dependent signal from the striatum (O’Doherty et al., 2003; Tanaka et al., 2004; Daw et al., 2006; Hampton et al., 2006) both show activity patterns that are analogous to TD errors.”
BackgroundSelf-citationReward and the serotonergic system
Kranz, G. S., Kasper, S., & Lanzenberger, R. (2010) · Neuroscience · doi:10.1016/j.neuroscience.2010.01.036 · Cited by 267
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277
“We fit participants’ choices using a temporal-difference learning model (Sutton and Barto, 1998) similar to those used to account for choice behavior and neural signals in previous studies (Daw et al., 2006; Li et al., 2006; Tanaka et al., 2004).”
BackgroundMethodInfluentialSucrose activates human taste pathways differently from artificial sweetener
Frank, G. K. W., Oberndorfer, T. A., Simmons, A. N., Paulus, M. P., Fudge, J. L., Yang, T. T., & Kaye, W. H. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.10.061 · Cited by 274
“It is associated with immediate reward processing (Tanaka et al., 2004) and prediction of future rewards (O'Doherty, 2004), while the anterior cingulate processes error monitoring and anticipation of reward (Carter et al.”
Multilocus Genetic Profile for Dopamine Signaling Predicts Ventral Striatum Reactivity
Nikolova, Y. S., Ferrell, R. E., Manuck, S. B., & Hariri, A. R. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2011.82 · Cited by 269
Plichta, M. M., Vasic, N., Wolf, R. C., Lesch, K.-P., Brummer, D., Jacob, C., Fallgatter, A. J., & Grön, G. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.07.008 · Cited by 272
Christakou, A., Brammer, M., & Rubia, K. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.08.067 · Cited by 266
Activation of Dorsal Raphe Serotonin Neurons Underlies Waiting for Delayed Rewards
Miyazaki, K., Miyazaki, K. W., & Doya, K. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3714-10.2011 · Cited by 256
“Human functional magnetic resonance imaging studies showed brain activity correlated with the values of delayed rewards (McClure et al., 2004; Tanaka et al., 2004, 2006).”
Self-citationLow-Serotonin Levels Increase Delayed Reward Discounting in Humans
Schweighofer, N., Bertin, M., Shishida, K., Okamoto, Y., Tanaka, S. C., Yamawaki, S., & Doya, K. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4982-07.2008 · Cited by 260
“Reinforcement learning models offer a biologically plausible framework to model human delayed reward choices (Pagnoni et al., 2002; O’Doherty et al., 2004; Tanaka et al., 2004, 2007).”
BackgroundResultSelf-citationHuman substantia nigra neurons encode unexpected financial rewards
Zaghloul, K. A., Blanco, J. A., Weidemann, C. T., McGill, K., Jaggi, J. L., Baltuch, G. H., & Kahana, M. J. (2009) · Science · doi:10.1126/science.1167342 · Cited by 264
Wittmann, M., Leland, D. S., & Paulus, M. P. (2007) · Experimental Brain Research · doi:10.1007/s00221-006-0822-y · Cited by 255
Serotonin, inhibition, and negative mood
Dayan, P., & Huys, Q. J. M. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0040004 · Cited by 263
Rogers, R. D. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.165 · Cited by 249
“In humans, (Tanaka et al (2004, 2007) have articulated a model in which serotonin is proposed to modulate reinforcement value signals across different time scales within the ventral striatum and the dorsal striatum, respectively.”
BackgroundDelParigi, A., Chen, K., Salbe, A. D., Hill, J. O., Wing, R. R., Reiman, E. M., & Tataranni, P. A. (2007) · International Journal of Obesity · doi:10.1038/sj.ijo.0803431 · Cited by 255
Prefrontal Coding of Temporally Discounted Values during Intertemporal Choice
Kim, S., Hwang, J., & Lee, D. (2008) · Neuron · doi:10.1016/j.neuron.2008.05.010 · Cited by 239
Neurocomputational models of basal ganglia function in learning, memory and choice
Cohen, M. X., & Frank, M. J. (2009) · Behavioural Brain Research · doi:10.1016/j.bbr.2008.09.029 · Cited by 237
Reward-Dependent Modulation of Neuronal Activity in the Primate Dorsal Raphe Nucleus
Nakamura, K., Matsumoto, M., & Hikosaka, O. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.0021-08.2008 · Cited by 245
“Recent studies have suggested that multiple neural systems may participate in the representation of rewards at different timescales (McClure et al., 2004; Tanaka et al., 2004).”
BackgroundLiu, X., Powell, D. K., Wang, H., Gold, B. T., Corbly, C. R., & Joseph, J. E. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5227-06.2007 · Cited by 241
The neurobiology of punishment
Seymour, B., Singer, T., & Dolan, R. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2119 · Cited by 240
Calculating utility: preclinical evidence for cost–benefit analysis by mesolimbic dopamine
Phillips, P. E. M., Walton, M. E., & Jhou, T. C. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0626-6 · Cited by 237
“…linking an action with an immediately available outcome but not when learning to obtain a large future reward by overcoming small, immediate losses (Tanaka et al. 2004) and another found rostral anterior cingulate/ medial orbitofrontal cortex and ventral striatum activations when choosing to…”
Background
Metadata
- Authors
- Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Japan Science and Technology Agency (JP)
- Hiroshima University (JP)
- Corresponding author
- Doya, Kenji
- Venue
- Nature Neuroscience
- Published
- 2004-07-04
- Volume
- 7
- Issue
- 8
- Pages
- 887-893
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- OpenAlex
- W2032610016
- Cited by
- 779
- References
- 54
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 9 · 2022: 29 · 2021: 28 · 2020: 32 · 2019: 25 · 2018: 32 · 2017: 37 · 2016: 40 · 2015: 45 · 2014: 50 · 2013: 42 · 2012: 47
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- reward prediction, time scales, cortico-basal ganglia loop, prediction error, insula, striatum, orbitofrontal cortex, dorsolateral prefrontal cortex, dorsal raphe nucleus, cerebellum, computational modeling, reinforcement learning
- MeSH terms
- Adult, Brain, Brain Mapping, Discrimination Learning, Female, Humans, Magnetic Resonance Imaging, Male, Markov Chains, Middle Aged, Reward
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.