The Successor Representation and Temporal Context
The DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index.
Abstract
The successor representation was introduced into reinforcement learning by Dayan ( 1993 ) as a means of facilitating generalization between states with similar successors. Although reinforcement learning in general has been used extensively as a model of psychological and neural processes, the psychological validity of the successor representation has yet to be explored. An interesting possibility is that the successor representation can be used not only for reinforcement learning but for episodic learning as well. Our main contribution is to show that a variant of the temporal context model (TCM; Howard & Kahana, 2002 ), an influential model of episodic memory, can be understood as directly estimating the successor representation using the temporal difference learning algorithm (Sutton & Barto, 1998 ). This insight leads to a generalization of TCM and new experimental predictions. In addition to casting a new normative light on TCM, this equivalence suggests a previously unexplored point of contact between different learning systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Gershman et al. (2012) showed that this new variant of TCM can be understood as directly estimating the SR using TD learning, with temporal context serving as an eligibility trace.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al. (2012)
In the citing paper
“During encoding (Figure 9E, top), incoming feature vectors φ(st) update a slowly drifting context ct (Eq. 76). This context vector serves as an eligibility trace in a TD update rule (Eq. 78) that learns the SR estimate M̂”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al., 2012
In the citing paper
“If the system is given access to appropriate memory stores and representations (Buonomano and Merzenich, 1995; Gershman et al., 2012, 2014) of temporal context, this could potentially mitigate the need for temporal credit assignment as such”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Gershman et al., 2012
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 5×
“An important—and computationally tractable—class of such learning problems arises when the environmental dynamics are specified by a Markov chain (a key assumption in RL theory; see Sutton & Barto, 1998), and the property to be inferred has a particular functional form that we describe below.”
InfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,661
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“Here we consider a temporal difference (TD) learning algorithm (Sutton, 1988) that estimates M directly from sample paths (see White, 1995, for more information).”
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 1×
“Sutton’s work served as the precursor to later theories of hierarchical RL, in which values can be defined over temporally abstract sequences of states and actions (Sutton et al., 1999).”
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
Kemeny, J. G., & Snell, J. L. (1976) · Medical Entomology and Zoology · Cited by 2,813
Research works printed but not linked: 7
- 10.1037/0278-7393.25.4.923 · doi:10.1037/0278-7393.25.4.923 · Paper page
- Mahadevan S., 2007, Journal of Machine Learning Research, 8, 2169
- 10.1037/0033-295X.95.4.434 · doi:10.1037/0033-295x.95.4.434 · Paper page
- Schank R., 1982, Dynamic memory
- Sederberg P., 2010, Psychonomic Bulletin and Review, 38, 689
- Socher R., 2009, Advances in neural information processing systems, 22, 1714
- 10.1037/0882-7974.13.2.256 · doi:10.1037/0882-7974.13.2.256 · Paper page
Cited by
Items 101–116 of 116
Associative Learning of an Unnormalized Successor Representation
Verosky, N. J. (2024) · Neural Computation · doi:10.1162/neco_a_01675 · Cited by 1
“…representation can be estimated with a TD learning rule that updates the estimate stored in matrix (cid:5) M SR at every time step (Dayan, 1993; Gershman et al., 2012; Stachen-feld et al., 2017; Sutton & Barto, 2018), where e n ( i ) is the “eligibility” of state i for updating at time step n…”
BackgroundMethodInfluentialPolicy-Oriented Cognitive Risk Map Modeling for Lane Change via Deep Successor Representation
Chen, D., Lu, C., Liu, Y., Meng, X., & Gong, J. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3533561 · Cited by 1
“It has been demonstrated that SR can efficiently capture essential features of rodent hippocampal location cells and model cognition within abstract conceptual spaces [23], [24].”
Sleep strengthens successor representations of learned sequences in humans
He, X., Büchel, P. K., Faghel-Soubeyrand, S., Klingspohr, J., Kehl, M. S., & Staresina, B. P. (2026) · PLoS Biology · doi:10.1371/journal.pbio.3003740 · Cited by 1
“More recent work has suggested that this framework may also apply to episodic memory (Gershman, 2012, 2018; Momennejad et al., 2017; Stachenfeld et al., 2017).”
Computational Modeling of Basic Cognitive Functionalities
(2023) · Cambridge University Press eBooks · doi:10.1017/9781108755610.014 · Cited by 1
Opponent Learning with Different Representations in the Cortico-Basal Ganglia Circuits
Morita, K., Shimomura, K., & Kawaguchi, Y. (2023) · eNeuro · doi:10.1523/eneuro.0422-22.2023 · Cited by 1
“These feature variables were updated through TD learning of SR features (Gardner et al., 2018; Gershman et al., 2012).”
MethodEpisodic memory and the temporal dynamics of cognitive controlPreprint
Wientjes, S., & Holroyd, C. B. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9jr52 · Cited by 2
Reinstatement of temporal context retrieves cognitive control demandsPreprint
Wientjes, S., & Holroyd, C. B. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/qgfz6 · Cited by 1
zarfati, S. B. M. (2023) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/a73vr
Computational Models of Hippocampal Cognitive Function
Bush, D., & Burgess, N. (2025) · Oxford University Press eBooks · doi:10.1093/med/9780190065324.003.0016
Towards a unifying theory of generalizationPreprint
Schulz, E. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/rzj2m_v1
Recurrent Neural Networks models : inference, representations, and the role of regularization
Fanthomme, A. (2021) · theses.fr (ABES) · doi:10.70675/afd73818z7545z4451zb9a3zd32aa98cd0f6
Distinct hippocampal codes emerge during brain-machine interface navigationPreprint
Micou, C., Ho, H., O’Leary, T., & Krupic, J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.11.724143
Kaşgöz, E., Gunn, J., & Talmi, D. (2026) · Learning & Behavior · doi:10.3758/s13420-026-00724-4
Hippocampal-Dependent Navigation without Outcome-Sensitive Control
Wang, S., Grgurich, R., & Blair, H. T. (2026) · Journal of Neuroscience · doi:10.1523/jneurosci.0389-26.2026
Metadata
- Authors
- Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B.
- Affiliations
- Princeton University (US)
- The Ohio State University (US)
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2012-06
- Volume
- 24
- Issue
- 6
- Pages
- 1553-1568
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076546217
- Cited by
- 117
- References
- 61
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 12 · 2021: 10 · 2020: 11 · 2019: 7 · 2018: 8 · 2017: 12 · 2016: 6 · 2014: 4 · 2013: 3 · 2012: 1
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- successor representation, temporal difference learning, episodic memory, episodic learning, reinforcement learning, normative model, memory generalization, learning systems
- MeSH terms
- Algorithms, Learning, Models, Psychological, Time Factors, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.