The Successor Representation and Temporal Context
The DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index.
Abstract
The successor representation was introduced into reinforcement learning by Dayan ( 1993 ) as a means of facilitating generalization between states with similar successors. Although reinforcement learning in general has been used extensively as a model of psychological and neural processes, the psychological validity of the successor representation has yet to be explored. An interesting possibility is that the successor representation can be used not only for reinforcement learning but for episodic learning as well. Our main contribution is to show that a variant of the temporal context model (TCM; Howard & Kahana, 2002 ), an influential model of episodic memory, can be understood as directly estimating the successor representation using the temporal difference learning algorithm (Sutton & Barto, 1998 ). This insight leads to a generalization of TCM and new experimental predictions. In addition to casting a new normative light on TCM, this equivalence suggests a previously unexplored point of contact between different learning systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Gershman et al. (2012) showed that this new variant of TCM can be understood as directly estimating the SR using TD learning, with temporal context serving as an eligibility trace.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al. (2012)
In the citing paper
“During encoding (Figure 9E, top), incoming feature vectors φ(st) update a slowly drifting context ct (Eq. 76). This context vector serves as an eligibility trace in a TD update rule (Eq. 78) that learns the SR estimate M̂”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al., 2012
In the citing paper
“If the system is given access to appropriate memory stores and representations (Buonomano and Merzenich, 1995; Gershman et al., 2012, 2014) of temporal context, this could potentially mitigate the need for temporal credit assignment as such”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Gershman et al., 2012
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 5×
“An important—and computationally tractable—class of such learning problems arises when the environmental dynamics are specified by a Markov chain (a key assumption in RL theory; see Sutton & Barto, 1998), and the property to be inferred has a particular functional form that we describe below.”
InfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,461
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“Here we consider a temporal difference (TD) learning algorithm (Sutton, 1988) that estimates M directly from sample paths (see White, 1995, for more information).”
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 1×
“Sutton’s work served as the precursor to later theories of hierarchical RL, in which values can be defined over temporally abstract sequences of states and actions (Sutton et al., 1999).”
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
Kemeny, J. G., & Snell, J. L. (1976) · Medical Entomology and Zoology · Cited by 2,813
Research works printed but not linked: 7
- 10.1037/0278-7393.25.4.923 · doi:10.1037/0278-7393.25.4.923 · Paper page
- Mahadevan S., 2007, Journal of Machine Learning Research, 8, 2169
- 10.1037/0033-295X.95.4.434 · doi:10.1037/0033-295x.95.4.434 · Paper page
- Schank R., 1982, Dynamic memory
- Sederberg P., 2010, Psychonomic Bulletin and Review, 38, 689
- Socher R., 2009, Advances in neural information processing systems, 22, 1714
- 10.1037/0882-7974.13.2.256 · doi:10.1037/0882-7974.13.2.256 · Paper page
Cited by
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
“sentation, which represents a value function in terms of the expected future state occupancy. Decomposition of the successor representation yields reasonable sub-goals for spatial navigation problems [5, 12, 44]. Botvinick et al.[3] have written a general overview of hierarchical reinforcement learning in the context…”
BackgroundThe successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
“Using these updated representations, we compute Q(sa) at the end of phase 1 and again at the end of phase 2 using equation (5) for the SR learner and equation (6) for the MB learner.”
MethodSelf-citationPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
“In line with this, it has been noted that there is a close correspondence between the update rule used by SR-TD and update rules in the temporal context model of memory [19].”
MethodSelf-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
Self-citation
Metadata
- Authors
- Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B.
- Affiliations
- Princeton University (US)
- The Ohio State University (US)
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2012-06
- Volume
- 24
- Issue
- 6
- Pages
- 1553-1568
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076546217
- Cited by
- 117
- References
- 61
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 12 · 2021: 10 · 2020: 11 · 2019: 7 · 2018: 8 · 2017: 12 · 2016: 6 · 2014: 4 · 2013: 3 · 2012: 1
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- successor representation, temporal difference learning, episodic memory, episodic learning, reinforcement learning, normative model, memory generalization, learning systems
- MeSH terms
- Algorithms, Learning, Models, Psychological, Time Factors, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Successor Representation and Temporal Context. https://app.cochat.ai/papers/10.1162/neco_a_00282
- The paper
- Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012). The Successor Representation and Temporal Context. Neural Computation. https://doi.org/10.1162/neco_a_00282
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.