The Successor Representation and Temporal Context
The DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index.
Abstract
The successor representation was introduced into reinforcement learning by Dayan ( 1993 ) as a means of facilitating generalization between states with similar successors. Although reinforcement learning in general has been used extensively as a model of psychological and neural processes, the psychological validity of the successor representation has yet to be explored. An interesting possibility is that the successor representation can be used not only for reinforcement learning but for episodic learning as well. Our main contribution is to show that a variant of the temporal context model (TCM; Howard & Kahana, 2002 ), an influential model of episodic memory, can be understood as directly estimating the successor representation using the temporal difference learning algorithm (Sutton & Barto, 1998 ). This insight leads to a generalization of TCM and new experimental predictions. In addition to casting a new normative light on TCM, this equivalence suggests a previously unexplored point of contact between different learning systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Gershman et al. (2012) showed that this new variant of TCM can be understood as directly estimating the SR using TD learning, with temporal context serving as an eligibility trace.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al. (2012)
In the citing paper
“During encoding (Figure 9E, top), incoming feature vectors φ(st) update a slowly drifting context ct (Eq. 76). This context vector serves as an eligibility trace in a TD update rule (Eq. 78) that learns the SR estimate M̂”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al., 2012
In the citing paper
“If the system is given access to appropriate memory stores and representations (Buonomano and Merzenich, 1995; Gershman et al., 2012, 2014) of temporal context, this could potentially mitigate the need for temporal credit assignment as such”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Gershman et al., 2012
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 5×
“An important—and computationally tractable—class of such learning problems arises when the environmental dynamics are specified by a Markov chain (a key assumption in RL theory; see Sutton & Barto, 1998), and the property to be inferred has a particular functional form that we describe below.”
InfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,459
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“Here we consider a temporal difference (TD) learning algorithm (Sutton, 1988) that estimates M directly from sample paths (see White, 1995, for more information).”
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 1×
“Sutton’s work served as the precursor to later theories of hierarchical RL, in which values can be defined over temporally abstract sequences of states and actions (Sutton et al., 1999).”
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
Kemeny, J. G., & Snell, J. L. (1976) · Medical Entomology and Zoology · Cited by 2,813
Remembering the past to imagine the future: the prospective brain
Schacter, D. L., Addis, D. R., & Buckner, R. L. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2213 · Cited by 2,394 · cited 1×
“The idea that memories are based (at least in part) on predictions about the future resonates with recent neuroscientific work suggesting a close connection between remembering the past and envisioning the future (Schacter et al., 2007; Hassabis & Maguire, 2009).”
The time course of perceptual choice: The leaky, competing accumulator model.
Usher, M., & McClelland, J. L. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.550 · Cited by 2,152 · cited 2×
“(2008) simulations is that we replaced the leaky competitive accumulator (Usher and McClelland, 2001) with the linear ballistic accumulator (LBA; Brown and Heathcote, 2008) for computational speed.”
Addis, D. R., Wong, A. T., & Schacter, D. L. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.10.016 · Cited by 1,949 · cited 1×
“active as when subjects recall autobiographical memories (Addis et al., 2007; Szpunar et al., 2007).”
Incremental interpretation at verbs: restricting the domain of subsequent reference
Altmann, G. T. M., & Kamide, Y. (1999) · Cognition · doi:10.1016/s0010-0277(99)00059-1 · Cited by 1,735 · cited 2×
“Similarly, there is a sense in which understanding natural language involves predicting future words: Evidence suggests that people anticipate long-distance dependencies between words (Altmann and Kamide, 1999; Kamide et al., 2003).”
Patients with hippocampal amnesia cannot imagine new experiences
Hassabis, D., Kumaran, D., Vann, S. D., & Maguire, E. A. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0610561104 · Cited by 1,414 · cited 1×
“Damage to this network, particularly the hippocampus, impairs both autobiographical memory and episodic future thinking (Hassabis et al., 2007).”
Atance, C. M., & O'Neill, D. K. (2001) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01804-0 · Cited by 1,300 · cited 1×
“Perhaps less obviously, episodic memory also involves a predictive component: Plans and scripts retrieved from memory represent predictions about future events (Schank, 1982; Atance and O’Neill, 2001; Schacter et al., 2007).”
A Distributed Representation of Temporal Context
Howard, M. W., & Kahana, M. J. (2002) · Journal of Mathematical Psychology · doi:10.1006/jmps.2001.1388 · Cited by 1,174 · cited 5×
“In TCM, the context vector is used to cue memory retrieval, such that items experienced in a similar context are more likely to be retrieved; we will not discuss the recall operations further in this letter (see Howard & Kahana, 2002; Sederberg, Howard, & Kahana, 2008).”
InfluentialGläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
The simplest complete model of choice response time: Linear ballistic accumulation
Brown, S. D., & Heathcote, A. (2008) · Cognitive Psychology · doi:10.1016/j.cogpsych.2007.12.002 · Cited by 1,123 · cited 2×
“(2008) simulations is that we replaced the leaky competitive accumulator (Usher and McClelland, 2001) with the linear ballistic accumulator (LBA; Brown and Heathcote, 2008) for computational speed.”
Remembering the past and imagining the future: A neural model of spatial memory and imagery.
Byrne, P., Becker, S., & Burgess, N. (2007) · Psychological Review · doi:10.1037/0033-295x.114.2.340 · Cited by 1,011 · cited 1×
“A number of computational theories have attempted to explain how episodic memory and prediction arise from the same hippocampal circuitry (e.g., Levy et al., 2005; Byrne et al., 2007); of particular relevance is the work of Zilli and Hasselmo (2008), who showed how episodic memories stored in the hippocampus could fac…”
Kamide, Y., Altmann, G. T. M., & Haywood, S. L. (2003) · Journal of Memory and Language · doi:10.1016/s0749-596x(03)00023-8 · Cited by 959 · cited 1×
“Similarly, there is a sense in which understanding natural language involves predicting future words: Evidence suggests that people anticipate long-distance dependencies between words (Altmann and Kamide, 1999; Kamide et al., 2003).”
Recognition and retrieval processes in free recall.
Anderson, J. R., & Bower, G. H. (1972) · Psychological Review · doi:10.1037/h0033773 · Cited by 820 · cited 1×
“This conceptualization of temporal context as a recency-weighted average of experienced items distinguishes TCM from earlier models of memory that posited randomly fluctuating context representations (Estes, 1955; Anderson & Bower, 1972; Mensink & Raaijmakers, 1988).”
A temporal ratio model of memory
Brown, G. D. A., Neath, I., & Chater, N. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.539 · Cited by 798 · cited 1×
“This explanation of recency effects is consistent with the principle of temporal distinctiveness posited by many previous models (e.g., Glenberg and Swanson, 1986; Brown et al., 2007): recall of an item depends on its relative recency to other list items, not its absolute recency.”
Neural substrates of envisioning the future
Szpunar, K. K., Watson, J. M., & McDermott, K. B. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0610082104 · Cited by 784 · cited 1×
“active as when subjects recall autobiographical memories (Addis et al., 2007; Szpunar et al., 2007).”
Neural organization for the long-term memory of paired associates
Sakai, K., & Miyashita, Y. (1991) · Nature · doi:10.1038/354152a0 · Cited by 743 · cited 2×
“For example, Sakai and Miyashita ( 1991 ) recorded cells in the anterior temporal cortex that preferentially responded to one stimulus and gradually increased their firing rate during a delay interval when cued with a paired associate of the preferred stimulus.”
A context maintenance and retrieval model of organizational processes in free recall
Polyn, S. M., Norman, K. A., & Kahana, M. J. (2009) · Psychological Review · doi:10.1037/a0014420 · Cited by 726 · cited 1×
“In particular, we draw a formal connection between TD learning of the SR and an influential model of episodic memory, the Temporal Context Model (TCM; Howard and Kahana, 2002; Howard et al., 2005; Sederberg et al., 2008; Polyn et al., 2009; Socher et al., 2009), resulting in a generalized form of TCM.”
Self-citationMetalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704 · cited 2×
“While the TD-SR algorithm provides insight into the normative rationale of TCM, it does not offer guidance in the setting of parameters, since λ and γ are considered idiosyncratic properties of the agent (but see Doya, 2002 ).”
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · cited 5×
“Starting from a generic prediction problem in Markov chains, we presented a learning algorithm (TD-SR) for solving this problem (see also Dayan, 1993; White, 1995) and then established its equivalence to TCM, a well-known model of episodic memory.”
InfluentialHierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646 · cited 1×
“3 Because”
Learning and Memory: A Comprehensive Reference
(2008) · Elsevier eBooks · doi:10.1016/c2009-1-03733-2 · Cited by 589
Associative retrieval processes in free recall
Kahana, M. J. (1996) · Memory & Cognition · doi:10.3758/bf03197276 · Cited by 583 · cited 3×
“In particular, Kahana (1996) measured recall contingencies: if the previously recalled item was studied at serial position j, what is the probability that the next-recalled item will be from serial position j+lag?”
Hebbian Synapses: Biophysical Mechanisms and Algorithms
Brown, T. H., Kairiss, E. W., & Keenan, C. L. (1990) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.13.030190.002355 · Cited by 558 · cited 1×
“It is worth noting that, from a neurobiological point of view, Hebbian learning of this sort can lead to instabilities that have disastrous consequences for a memory system (Brown et al., 1990).”
Vinogradova, O. S. (2001) · Hippocampus · doi:10.1002/hipo.1073 · Cited by 548 · cited 1×
“This view of CA1 as the locus of prediction error computation is consistent with earlier proposals characterizing CA1 as a " comparator " whose output represents a novelty (mismatch) signal (Lisman & Otmakhova, 2001; Vinogradova, 2001; Kumaran & Maguire, 2007).”
Statistical theory of spontaneous recovery and regression
Estes, W. K. (1955) · Psychological Review · doi:10.1037/h0048509 · Cited by 511 · cited 1×
“This conceptualization of temporal context as a recency-weighted average of experienced items distinguishes TCM from earlier models of memory that posited randomly fluctuating context representations (Estes, 1955; Anderson & Bower, 1972; Mensink & Raaijmakers, 1988).”
The construction system of the brain
Hassabis, D., & Maguire, E. A. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0296 · Cited by 483 · cited 2×
“The idea that memories are based (at least in part) on predictions about the future resonates with recent neuroscientific work suggesting a close connection between remembering the past and envisioning the future (Schacter et al., 2007; Hassabis and Maguire, 2009).”
Contextual variability and serial position effects in free recall.
Howard, M. W., & Kahana, M. J. (1999) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.25.4.923 · Cited by 469 · cited 1×
“For example, the recency effect refers to the observation that subjects tend to initiate recall with items that were presented at the end of the study list (Deese & Kaufman, 1957); interposing a distractor task between study and test dramatically reduces this effect (Howard & Kahana, 1999).”
A model for interference and forgetting.
Mensink, G.-J., & Raaijmakers, J. G. (1988) · Psychological Review · doi:10.1037//0033-295x.95.4.434 · Cited by 461 · cited 1×
“This conceptualization of temporal context as a recency-weighted average of experienced items distinguishes TCM from earlier models of memory that posited randomly fluctuating context representations (Estes, 1955; Anderson & Bower, 1972; Mensink & Raaijmakers, 1988).”
Prospective coding for objects in primate prefrontal cortex
Rainer, G., Rao, S. C., & Miller, E. K. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-13-05493.1999 · Cited by 432 · cited 2×
“3 Because”
Serial effects in recall of unorganized and sequentially organized verbal material.
Deese, J., & Kaufman, R. A. (1957) · Journal of Experimental Psychology · doi:10.1037/h0040536 · Cited by 404 · cited 1×
“For example, the recency effect refers to the observation that subjects tend to initiate recall with items that were presented at the end of the study list (Deese & Kaufman, 1957); interposing a distractor task between study and test dramatically reduces this effect (Howard & Kahana, 1999).”
Lisman, J. E., & Otmakhova, N. A. (2001) · Hippocampus · doi:10.1002/hipo.1071 · Cited by 401 · cited 1×
“These observations are paralleled by findings from rodent physiology showing that hippocampal cells encode predictions about future spatial locations (Lisman & Redish, 2009).”
A context-based theory of recency and contiguity in free recall.
Sederberg, P. B., Howard, M. W., & Kahana, M. J. (2008) · Psychological Review · doi:10.1037/a0013396 · Cited by 383 · cited 4×
“See Sederberg et al. (2008) for details of the recall mechanism in TCM.”
InfluentialSelf-citationMahadevan, S., & Maggioni, M. (2007) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.5555/1314498.1314570 · Cited by 374 · cited 1×
“See Mahadevan and Maggioni (2007) for discussion in the context of RL.”
Prediction, sequences and the hippocampus
Lisman, J., & Redish, A. D. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0316 · Cited by 287 · cited 2×
“These observations are paralleled by findings from rodent physiology showing that hippocampal cells encode predictions about future spatial locations (Lisman & Redish, 2009).”
Howard, M. W., Fotedar, M. S., Datey, A. V., & Hasselmo, M. E. (2005) · Psychological Review · doi:10.1037/0033-295x.112.1.75 · Cited by 283 · cited 2×
“Howard et al. (2005) have proposed a detailed mapping of TCM onto medial temporal lobe structures (including the hippocampus ); given the equivalence between (generalized) TCM and TD-SR, it may be sufficient simply to replace the Hebbian learning rule Howard et al. (2005) used to model plasticity…”
A temporal distinctiveness theory of recency and modality effects.
Glenberg, A. M., & Swanson, N. G. (1986) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.12.1.3 · Cited by 269 · cited 2×
“This explanation of recency effects is consistent with the principle of temporal distinctiveness posited by many previous models (Glenberg & Swanson, 1986; Brown, Neath, & Chater, 2007): recall of an item depends on its relative recency to other list items, not its absolute recency.”
Entorhinal-hippocampal connections: a speculative view of their function
Jones, R. S. G. (1993) · Trends in Neurosciences · doi:10.1016/0166-2236(93)90018-h · Cited by 259 · cited 7×
“EC inputs should not be purely sensory; they should partially reflect future successor predictions (the Msn+1 j term in the prediction error), which might be realized through feedback from CA3 to EC (Jones, 1993).”
InfluentialWhich computational mechanisms operate in the hippocampus during novelty detection?
Kumaran, D., & Maguire, E. A. (2007) · Hippocampus · doi:10.1002/hipo.20326 · Cited by 196 · cited 2×
“This view of CA1 as the locus of prediction error computation is consistent with earlier proposals characterizing CA1 as a “comparator” whose output represents a novelty (mismatch) signal (e.g., Lisman and Otmakhova, 2001; Vinogradova, 2001; Kumaran and Maguire, 2007).”
A Neural Signature of Hierarchical Reinforcement Learning
Ribas-Fernandes, J. J. F., Solway, A., Diuk, C., McGuire, J. T., Barto, A. G., Niv, Y., & Botvinick, M. M. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.042 · Cited by 195 · cited 1×
“Recent studies have suggested that such temporal abstractions operate in the brain during decision-making tasks (Botvinick, Niv, & Barto, 2009; Ribas-Fernandes et al., 2011).”
The temporal contiguity effect predicts episodic memory performance
Sederberg, P. B., Miller, J. F., Howard, M. W., & Kahana, M. J. (2010) · Memory & Cognition · doi:10.3758/mc.38.6.689 · Cited by 174 · cited 1×
“(Reproduced from Sederberg et al., 2010 .) back in time to the moment when the just-retrieved item was studied.”
Self-citationTD Models: Modeling the World at a Mixture of Time Scales
Sutton, R. S. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50072-4 · Cited by 145 · cited 1×
“For example, Sutton (1995) observed that the SR learns a representation of states at a single timescale; he proposed a generalization in which learning occurs at multiple timescales by mixing together models with different temporal horizons.”
Unsworth, N. (2007) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.33.6.1020 · Cited by 130 · cited 1×
“Experimentally, a number of individual differences have been correlated with these parameters, such as age (Wingfield, Lindfield, & Kahana, 1998) and working memory capacity (Unsworth, 2007).”
Associative Retrieval Processes in Episodic Memory
Kahana, M. J., Howard, M. W., & Polyn, S. M. (2008) · Learning and Memory: A Comprehensive Reference · doi:10.1016/b978-012370509-9.00185-6 · Cited by 120 · cited 1×
“We first briefly review some of the basic empirical findings from free recall experiments motivating TCM (for a comprehensive review, see Kahana, Howard, & Polyn, 2008 ).”
Spacing and lag effects in free recall of pure lists
Kahana, M. J., & Howard, M. W. (2005) · Psychonomic Bulletin & Review · doi:10.3758/bf03196362 · Cited by 104 · cited 2×
“Other free recall studies have presented items repeatedly at study (e.g., to investigate spacing effects; Kahana and Howard, 2005), but none of these studies manipulated successor relationships in a manner that would allow us to tease apart the models.”
Modeling the role of working memory and episodic memory in behavioral tasks
Zilli, E. A., & Hasselmo, M. E. (2008) · Hippocampus · doi:10.1002/hipo.20382 · Cited by 77 · cited 1×
“…arise from the same hippocampal circuitry (Levy, Hocking, & Wu, 2005; Byrne, Becker, & Burgess, 2007); of particular relevance is the work of Zilli and Hasselmo (2008), who showed how episodic memories stored in the hippocampus could facilitate long-term reward prediction for Markov chains.”
Adult age differences in the temporal characteristics of category free recall.
Wingfield, A., Lindfield, K. C., & Kahana, M. J. (1998) · Psychology and Aging · doi:10.1037//0882-7974.13.2.256 · Cited by 65
Bridging the gap: Transitive associations between items presented in similar temporal contexts.
Howard, M. W., Jing, B., Rao, V. A., Provyn, J. P., & Datey, A. V. (2009) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0015002 · Cited by 60
Constructing Semantic Representations From a Gradually Changing Representation of Temporal Context
Howard, M. W., Shankar, K. H., & Jagadisan, U. K. K. (2011) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2010.01112.x · Cited by 57 · cited 1×
“TCM was originally designed to describe the associative processes that underlie episodic memory and applied to human behavior in free recall experiments (for extensions of the TCM framework to semantic learning, see Howard, Shankar, & Jagadisan, 2011 ; Shankar, Jagadisan, & Howard, 2009 ).”
Interpreting hippocampal function as recoding and forecasting
Levy, W. B., Hocking, A. B., & Wu, X. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.08.005 · Cited by 53 · cited 1×
“As described above, repeating the AB subsequence forms a strong association from A to B, but the Hebbian learning rule grows without bounds with each repetition, which gives rise to an associative strength from A → B that is much larger for the Hebbian than the TD learning rule.”
A Bayesian Analysis of Dynamics in Free Recall
Socher, R., Gershman, S. J., Sederberg, P. B., Norman, K. A., Perotte, A., & Blei, D. M. (2009) · neural information processing systems · Cited by 36
Self-citationSequential learning using temporal context
Shankar, K. H., Jagadisan, U. K. K., & Howard, M. W. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2009.07.005 · Cited by 16 · cited 1×
“TCM was originally designed to describe the associative processes that underlie episodic memory and applied to human behavior in free recall experiments (for extensions of the TCM framework to semantic learning, see Howard et al., 2011; Shankar et al., 2009).”
Research works printed but not linked: 7
- 10.1037/0278-7393.25.4.923 · doi:10.1037/0278-7393.25.4.923 · Paper page
- Mahadevan S., 2007, Journal of Machine Learning Research, 8, 2169
- 10.1037/0033-295X.95.4.434 · doi:10.1037/0033-295x.95.4.434 · Paper page
- Schank R., 1982, Dynamic memory
- Sederberg P., 2010, Psychonomic Bulletin and Review, 38, 689
- Socher R., 2009, Advances in neural information processing systems, 22, 1714
- 10.1037/0882-7974.13.2.256 · doi:10.1037/0882-7974.13.2.256 · Paper page
Cited by
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
“sentation, which represents a value function in terms of the expected future state occupancy. Decomposition of the successor representation yields reasonable sub-goals for spatial navigation problems [5, 12, 44]. Botvinick et al.[3] have written a general overview of hierarchical reinforcement learning in the context…”
BackgroundThe successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
“Using these updated representations, we compute Q(sa) at the end of phase 1 and again at the end of phase 2 using equation (5) for the SR learner and equation (6) for the MB learner.”
MethodSelf-citationPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
“In line with this, it has been noted that there is a close correspondence between the update rule used by SR-TD and update rules in the temporal context model of memory [19].”
MethodSelf-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
Self-citation
Metadata
- Authors
- Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B.
- Affiliations
- Princeton University (US)
- The Ohio State University (US)
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2012-06
- Volume
- 24
- Issue
- 6
- Pages
- 1553-1568
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076546217
- Cited by
- 117
- References
- 61
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 12 · 2021: 10 · 2020: 11 · 2019: 7 · 2018: 8 · 2017: 12 · 2016: 6 · 2014: 4 · 2013: 3 · 2012: 1
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- successor representation, temporal difference learning, episodic memory, episodic learning, reinforcement learning, normative model, memory generalization, learning systems
- MeSH terms
- Algorithms, Learning, Models, Psychological, Time Factors, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.