The DOI matches its Crossref record; 128 of the 128 references it cites were found in a registry or an index.
Predictive representations can link model-based reinforcement learning to model-free mechanisms
Open accessPublished version of a preprint
Abstract
Abstract Humans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence suggest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation. Author Summary According to standard models, when confronted with a choice, animals and humans rely on two separate, distinct processes to come to a decision. One process deliberatively evaluates the consequences of each candidate action and is thought to underlie the ability to flexibly come up with novel plans. The other process gradually increases the propensity to perform behaviors that were previously successful and is thought to underlie automatically executed, habitual reflexes. Although computational principles and animal behavior support this dichotomy, at the neural level, there is little evidence supporting a clean segregation. For instance, although dopamine — famously implicated in drug addiction and Parkinson’s disease — currently only has a well-defined role in the automatic process, evidence suggests that it also plays a role in the deliberative process. In this work, we present a computational framework for resolving this mismatch. We show that the types of behaviors associated with either process could result from a common learning mechanism applied to different strategies for how populations of neurons could represent candidate actions. In addition to demonstrating that this account can produce the full range of flexible behavior observed in the empirical literature, we suggest experiments that could detect the various approaches within this framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
RETRACTED: A mathematical model explains saturating axon guidance responses to molecular gradients
Nguyen, H., Dayan, P., Pujic, Z., Cooper-White, J., & Goodhill, G. J. (2016) · eLife · doi:10.7554/elife.12248 · Cited by 1,221
RetractedReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
(2014) · Climate Change 2013 – The Physical Science Basis · doi:10.1017/cbo9781107415324.004 · Cited by 23,880
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
(1978) · UA Campus Repository (The University of Arizona) · Cited by 8,010
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256
The hippocampus as a cognitive map
Passingham, R. E. (1979) · Neuroscience · doi:10.1016/0306-4522(79)90015-0 · Cited by 5,128
Functional architecture of basal ganglia circuits: neural substrates of parallel processing
Alexander, G. E., & Crutcher, M. D. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90107-l · Cited by 4,131
Reactivation of hippocampal ensemble memories during sleep
Wilson, M. A., & McNaughton, B. L. (1994) · Science · doi:10.1126/science.8036517 · Cited by 3,118
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Self-citationBalleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Two-stage model of memory trace formation: A role for “noisy” brain states
Buzsáki, G. (1989) · Neuroscience · doi:10.1016/0306-4522(89)90423-5 · Cited by 1,709
Selective suppression of hippocampal ripples impairs spatial memory
Girardeau, G., Benchenane, K., Wiener, S. I., Buzsáki, G., & Zugaro, M. B. (2009) · Nature Neuroscience · doi:10.1038/nn.2384 · Cited by 1,560
A century of trends in adult human height
NCD Risk Factor Collaboration (NCD-RisC) (2016) · eLife · doi:10.7554/elife.13410 · Cited by 1,558
Experience-weighted Attraction Learning in Normal Form Games
Camerer, C., & Hua Ho, T. (1999) · Econometrica · doi:10.1111/1468-0262.00054 · Cited by 1,487
The primate basal ganglia: Parallel and integrative networks
Haber, S. N. (2003) · Journal of Chemical Neuroanatomy · doi:10.1016/j.jchemneu.2003.10.003 · Cited by 1,479 · cited 1×
“Also contrary to a strict division between systems, both dorsomedial and dorsolateral striatal territories have similar interrelationships with dopamine [48], though the involvement of their differential dopaminergic input in devaluation sensitivity has not been completely assessed [49].”
BackgroundChurchill, A. J., Gutiérrez, G. D., Singer, R. A., Lorberbaum, D. S., Fischer, K. A., & Sussel, L. (2017) · eLife · doi:10.7554/elife.20010 · Cited by 1,236
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Self-citationTonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · cited 1×
“These effects may instead be related to functions of dopamine other than prediction error signaling, such as tonic dopamine’s involvement supporting working memory [69] or its hypothesized role controlling the allocation of cognitive effort [41,70,71].”
BackgroundSelf-citationThe role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationMotivational control of goal-directed action
Dickinson, A., & Balleine, B. (1994) · Animal Learning & Behavior · doi:10.3758/bf03199951 · Cited by 995
Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis
Glimcher, P. W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014269108 · Cited by 955
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Awake hippocampal sharp-wave ripples support spatial memory
Jadhav, S. P., Kemere, C., German, P. W., & Frank, L. M. (2012) · Science · doi:10.1126/science.1217230 · Cited by 924
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885
Disruption of ripple‐associated hippocampal activity during rest impairs spatial learning in the rat
Ego‐Stengel, V., & Wilson, M. A. (2010) · Hippocampus · doi:10.1002/hipo.20707 · Cited by 842
Prioritized sweeping: Reinforcement learning with less data and less time
Moore, A. W., & Atkeson, C. G. (1993) · Machine Learning · doi:10.1007/bf00993104 · Cited by 823
Animal Intelligence. Experimental Studies
Thorndike, E. L., & Jelliffe (1912) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-191205000-00016 · Cited by 817
Wolff, P., & Barbey, A. K. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00001 · Cited by 788
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777
Kudrimoti, H. S., Barnes, C. A., & McNaughton, B. L. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-10-04090.1999 · Cited by 753
What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?
Doya, K. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00046-5 · Cited by 747
Hippocampal replay is not a simple function of experience
Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S., & Redish, A. D. (2010) · Neuron · doi:10.1016/j.neuron.2010.01.034 · Cited by 733
Neurocomputational models of working memory
Durstewitz, D., Seamans, J. K., & Sejnowski, T. J. (2000) · Nature Neuroscience · doi:10.1038/81460 · Cited by 724
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693
Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646
Self-citationNeural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642
Human orbitofrontal cortex represents a cognitive map of state space
Schuck, N. W., Cai, M. B., Wilson, R. C., & Niv, Y. (2016) · Neuron · doi:10.1016/j.neuron.2016.08.019 · Cited by 602
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Self-citationNeural representations of events arise from temporal community structure
Schapiro, A. C., Rogers, T. T., Cordova, N. I., Turk-Browne, N. B., & Botvinick, M. M. (2013) · Nature Neuroscience · doi:10.1038/nn.3331 · Cited by 556
Self-citationHow robust is the language architecture? The case of mood
Van Berkum, J. J. A., De Goede, D., Van Alphen, P. M., Mulder, E. R., & Kerstholt, J. H. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00505 · Cited by 555
A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement
Houk, J. C., Adams, J. L., & Barto, A. G. (1994) · Models of Information Processing in the Basal Ganglia · doi:10.7551/mitpress/4708.003.0020 · Cited by 541
Preference by Association: How Memory Mechanisms in the Hippocampus Bias Decisions
Wimmer, G. E., & Shohamy, D. (2012) · Science · doi:10.1126/science.1223252 · Cited by 515
Computing the Local Field Potential (LFP) from Integrate-and-Fire Network Models
Mazzoni, A., Lindén, H., Cuntz, H., Lansner, A., Panzeri, S., & Einevoll, G. T. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004584 · Cited by 461
Introduction and removal of reward, and maze performance in rats.
Tolman, E. C., & Honzik, C. H. (1930) · Cited by 433
Predictive Representations of State
Littman, M. L., & Sutton, R. S. (2001) · Cited by 425
Value Representations in the Primate Striatum during Matching Behavior
Lau, B., & Glimcher, P. W. (2008) · Neuron · doi:10.1016/j.neuron.2008.02.021 · Cited by 409
Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation
Faure, A., Haberland, U., Condé, F., & Massioui, N. E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3894-04.2005 · Cited by 402
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Self-citationThe Role of Learning in the Operation of Motivational Systems
Dickinson, A., & Balleine, B. (2002) · Stevens' Handbook of Experimental Psychology · doi:10.1002/0471214426.pas0312 · Cited by 384
A map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex
Wikenheiser, A. M., & Schoenbaum, G. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.56 · Cited by 359
Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal target
Parker, N. F., Cameron, C. M., Taliaferro, J. P., Lee, J., Choi, J. Y., Davidson, T. J., Daw, N. D., & Witten, I. B. (2016) · Nature Neuroscience · doi:10.1038/nn.4287 · Cited by 355
Self-citationSpeed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · cited 3×
“In this regard, we expect that the prioritization of replay, like the arbitration between model-based vs model-free tradeoffs [1,41,79], might operate to the principles of efficient cost-benefit management.”
BackgroundMethodSuccessor Features for Transfer in Reinforcement Learning
Barreto, A. D. M. S., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Silver, D., & van Hasselt, H. P. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35439.18085 · Cited by 305
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297
Dopamine enhances model-based over model-free choice behavior
Wunderlich, K., Smittenaar, P., & Dolan, R. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.042 · Cited by 294
Model-based choices involve prospective neural activity
Doll, B. B., Duncan, K. D., Simon, D. A., Shohamy, D., & Daw, N. D. (2015) · Nature Neuroscience · doi:10.1038/nn.3981 · Cited by 283
Self-citationOnline evaluation of novel choices by simultaneous representation of multiple memories
Barron, H. C., Dolan, R. J., & Behrens, T. E. J. (2013) · Nature Neuroscience · doi:10.1038/nn.3515 · Cited by 278
Deserno, L., Huys, Q. J. M., Boehme, R., Buchert, R., Heinze, H.-J., Grace, A. A., Dolan, R. J., Heinz, A., & Schlagenhauf, F. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1417219112 · Cited by 233
Spike-Timing-Dependent Hebbian Plasticity as Temporal Difference Learning
Rao, R. P. N., & Sejnowski, T. J. (2001) · Neural Computation · doi:10.1162/089976601750541787 · Cited by 226
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217
Self-citationInterplay of approximate planning strategies
Huys, Q. J. M., Lally, N., Faulkner, P., Eshel, N., Seifritz, E., Gershman, S. J., Dayan, P., & Roiser, J. P. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1414219112 · Cited by 207
Self-citationAdaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum
Keramati, M., Smittenaar, P., Dolan, R. J., & Dayan, P. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1609094113 · Cited by 203
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195
Self-citationDeciding How To Decide: Self-Control and Meta-Decision Making
Boureau, Y.-L., Sokol-Hessner, P., & Daw, N. D. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.08.013 · Cited by 190
Self-citationSolway, A., & Botvinick, M. M. (2012) · Psychological Review · doi:10.1037/a0026435 · Cited by 182
Self-citationNeural Correlates of Forward Planning in a Spatial Decision Task in Humans
Simon, D. A., & Daw, N. D. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4647-10.2011 · Cited by 177
Self-citationAkam, T., Costa, R. M., & Dayan, P. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004648 · Cited by 172
Model-based hierarchical reinforcement learning and human action control
Botvinick, M., & Weinstein, A. (2014) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2013.0480 · Cited by 167
Self-citationPezzulo, G., Rigoli, F., & Chersi, F. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00092 · Cited by 163
Blundell, C., Uria, B., Pritzel, A., Li, Y., Ruderman, A., Leibo, J. Z., Rae, J., Wierstra, D., & Hassabis, D. (2016) · arXiv · doi:10.48550/arxiv.1606.04460 · Cited by 162
Temporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli
Nomoto, K., Schultz, W., Watanabe, T., & Sakagami, M. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4828-09.2010 · Cited by 158
Papale, A. E., Zielinski, M. C., Frank, L. M., Jadhav, S. P., & Redish, A. D. (2016) · Neuron · doi:10.1016/j.neuron.2016.10.028 · Cited by 152
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 146
TD Models: Modeling the World at a Mixture of Time Scales
Sutton, R. S. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50072-4 · Cited by 145
Gaussier, P., Revel, A., Banquet, J. P., & Babeau, V. (2002) · Biological Cybernetics · doi:10.1007/s004220100269 · Cited by 144
Wandering tales: evolutionary origins of mental time travel and language
Corballis, M. C. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00485 · Cited by 141
Khamassi, M., & Humphries, M. D. (2012) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2012.00079 · Cited by 141
Braun, E., Dittmar, L., Boeddeker, N., & Egelhaaf, M. (2012) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2012.00001 · Cited by 136
Multiple Forms of Value Learning and the Function of Dopamine
Balleine, B. W., Daw, N. D., & O’Doherty, J. P. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00024-5 · Cited by 135
Self-citationDopamine selectively remediates ‘model-based’ reward learning: a computational approach
Sharp, M. E., Foerde, K., Daw, N. D., & Shohamy, D. (2015) · Brain · doi:10.1093/brain/awv347 · Cited by 134
Self-citationDeficits in reflexive covert attention following cerebellar injury
Striemer, C. L., Cantelmi, D., Cusimano, M. D., Danckert, J. A., & Schweizer, T. A. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00428 · Cited by 130
Value Learning through Reinforcement
Daw, N. D., & Tobler, P. N. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00015-2 · Cited by 130
Self-citationHuman anterior prefrontal cortex encodes the ‘what’ and ‘when’ of future intentions
Momennejad, I., & Haynes, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.02.079 · Cited by 128
Self-citationPerception of biological motion in visual agnosia
Huberle, E., Rupek, P., Lappe, M., & Karnath, H. (2012) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2012.00056 · Cited by 127
Boorman, E. D., Rajendran, V. G., O’Reilly, J. X., & Behrens, T. E. J. (2016) · Neuron · doi:10.1016/j.neuron.2016.02.014 · Cited by 125
Variability in Dopamine Genes Dissociates Model-Based and Model-Free Reinforcement Learning
Doll, B. B., Bath, K. G., Daw, N. D., & Frank, M. J. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.1901-15.2016 · Cited by 120
Self-citationThe successor representation and temporal context
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012) · Neural Computation · doi:10.1162/neco_a_00282 · Cited by 117
Self-citationWikenheiser, A. M., Marrero-Garcia, Y., & Schoenbaum, G. M. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.003 · Cited by 117
Corbit, L. H., Ostlund, S. B., & Balleine, B. W. (2002) · Journal of Neuroscience · doi:10.1523/jneurosci.22-24-10976.2002 · Cited by 110
Solving the detour problem in navigation: a model of prefrontal and hippocampal interactions
Spiers, H. J., & Gilbert, S. J. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00125 · Cited by 94
Encoding of Prospective Tasks in the Human Prefrontal Cortex under Varying Task Loads
Momennejad, I., & Haynes, J. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.0492-13.2013 · Cited by 89
Self-citationDifferent CA1 and CA3 Representations of Novel Routes in a Shortcut Situation
Alvernhe, A., Cauter, T. V., Save, É., & Poucet, B. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1909-08.2008 · Cited by 85
Li, H., Watson, A., Olechwier, A. M., Anaya, M. A., Sorooshyari, S. K., Harnett, D. P., Lee, H.-K., Vielmetter, J., Fares, M. A., García, K. C., et al. (2017) · eLife · doi:10.7554/elife.28111 · Cited by 85
An Optimization Network for Matrix Inversion
Jang, J.-S., Lee, S.-Y., & Shin, S.-Y. (1987) · neural information processing systems · Cited by 82
Expectancies in decision making, reinforcement learning, and ventral striatum
van der Meer, M. A. A., & Redish, A. D. (2010) · Frontiers in Neuroscience · doi:10.3389/neuro.01.006.2010 · Cited by 81
Model-based action planning involves cortico-cerebellar and basal ganglia networks
Fermin, A. S. R., Yoshida, T., Yoshimoto, J., Ito, M., Tanaka, S., & Doya, K. (2016) · Scientific Reports · doi:10.1038/srep31378 · Cited by 79
Design Principles of the Hippocampal Cognitive Map
Stachenfeld, K., Botvinick, M., & Gershman, S. J. (2014) · Cited by 74
Self-citationGrid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning
Gustafson, N. J., & Daw, N. D. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002235 · Cited by 73
Self-citationRoute planning in cats, in relation to the visibility of the goal
Poucet, B., Thinus‐Blanc, C., & Chapuis, N. (1983) · Animal Behaviour · doi:10.1016/s0003-3472(83)80083-9 · Cited by 67
Horizontal biases in rats’ use of three-dimensional space
Jovalekic, A., Hayman, R. M. A., Bécares, N., Reid, H., Thomas, G. E. C., Wilson, J. J., & Jeffery, K. J. (2011) · Behavioural Brain Research · doi:10.1016/j.bbr.2011.02.035 · Cited by 64
Prediction of Immediate and Future Rewards Differentially Recruits Cortico-Basal Ganglia Loops
Tanaka, S., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2016) · Springer eBooks · doi:10.1007/978-4-431-55402-8_22 · Cited by 59
Hierarchical control of goal-directed action in the cortical–basal ganglia network
Balleine, B. W., Dezfouli, A., Ito, M., & Doya, K. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.06.001 · Cited by 54
Motivated Reinforcement Learning
Dayan, P. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0006 · Cited by 50
Winocur, G., Moscovitch, M., Rosenbaum, R. S., & Sekeres, M. (2010) · Hippocampus · doi:10.1002/hipo.20721 · Cited by 49
Advantages and Limitations of using Successor Features for Transfer in Reinforcement LearningPreprint
Lehnert, L., Tellex, S., & Littman, M. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.00102 · Cited by 38
The role of exploratory experience in a shortcut task by golden hamsters (Mesocricetus auratus)
Chapuis, N., Durup, M., & Thinus‐Blanc, C. (1987) · Animal Learning & Behavior · doi:10.3758/bf03204960 · Cited by 37
Dissociable contribution of prefrontal and striatal dopaminergic genes to learning in economic games
Set, E., Sáez, I., Zhu, L., Houser, D. E., Myung, N., Zhong, S., Ebstein, R. P., Chew, S. H., & Hsu, M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1316259111 · Cited by 37
Multiplexing signals in reinforcement learning with internal models and dopamine
Nakahara, H. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2014.01.001 · Cited by 34
Anticipatory responses of dopamine neurons and cortical neurons reproduced by internal model
Suri, R. E. (2001) · Experimental Brain Research · doi:10.1007/s002210100814 · Cited by 30
Dual Representations for Dynamic Programming and Reinforcement Learning
Wang, T., Bowling, M., & Schuurmans, D. (2007) · doi:10.1109/adprl.2007.368168 · Cited by 29
Associative Learning from Replayed ExperiencePreprint
Ludvig, E. A., Mirian, M. S., Kehoe, E. J., & Sutton, R. S. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/100800 · Cited by 23
Dayan, P. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0303 · Cited by 23
The Learning of World Models By Connectionist Networks
Sutton, R. S., & Pinette, B. (1985) · eScholarship (California Digital Library) · Cited by 14
The emotional consequences of being distracted
Barceló, F. (2009) · Frontiers in Neuroscience · doi:10.3389/neuro.01.010.2009 · Cited by 9
Tolman and Honzik (1930) revisited or the mazes of psychology (1930-1980)
Ciancia, F. (1991) · The Psychological Record · Cited by 9
Research works printed but not linked: 3
- BW Balleine, 2009, Neuroeconomics, Neuroeconomics, 367
- Schlegel M, White A, White M. Stable predictive representations with general value functions for continual learning. Continual Learning and Deep Networks workshop at the Neural Information Processing System Conference. 2017. Available: https://sites.ualberta.ca/~amw8/cldl.pdf
- AC Schapiro, 2013, Neural representations of events arise from temporal community structure, Nat Publ Gr, 16
Cited by
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationA map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Prioritized memory access explains planning and hippocampal replay
Mattar, M. G., & Daw, N. D. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0232-z · Cited by 393
Self-citationThe successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Self-citationSequential replay of nonspatial task states in the human hippocampus
Schuck, N. W., & Niv, Y. (2019) · Science · doi:10.1126/science.aaw5181 · Cited by 304
Metadata
- Authors
- Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-09-25
- Volume
- 13
- Issue
- 9
- Pages
- e1005768
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2950347959
- Cited by
- 326
- References
- 128
- FWCI
- 16.40
- Type
- journal article
- Language
- en
- Update notices
- new_version (2017-10-05)
- Open access
- gold
- Funders
- National Institute of Mental Health (US) (T32 MH019524)
- National Institutes of Health (5R01MH109177-02)
- John Templeton Foundation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.