The Successor Representation and Temporal Context
The DOI matches its Crossref record; 60 of the 61 references it cites were found in a registry or an index.
Abstract
The successor representation was introduced into reinforcement learning by Dayan ( 1993 ) as a means of facilitating generalization between states with similar successors. Although reinforcement learning in general has been used extensively as a model of psychological and neural processes, the psychological validity of the successor representation has yet to be explored. An interesting possibility is that the successor representation can be used not only for reinforcement learning but for episodic learning as well. Our main contribution is to show that a variant of the temporal context model (TCM; Howard & Kahana, 2002 ), an influential model of episodic memory, can be understood as directly estimating the successor representation using the temporal difference learning algorithm (Sutton & Barto, 1998 ). This insight leads to a generalization of TCM and new experimental predictions. In addition to casting a new normative light on TCM, this equivalence suggests a previously unexplored point of contact between different learning systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Gershman et al. (2012) showed that this new variant of TCM can be understood as directly estimating the SR using TD learning, with temporal context serving as an eligibility trace.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al. (2012)
In the citing paper
“During encoding (Figure 9E, top), incoming feature vectors φ(st) update a slowly drifting context ct (Eq. 76). This context vector serves as an eligibility trace in a TD update rule (Eq. 78) that learns the SR estimate M̂”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Gershman et al., 2012
In the citing paper
“If the system is given access to appropriate memory stores and representations (Buonomano and Merzenich, 1995; Gershman et al., 2012, 2014) of temporal context, this could potentially mitigate the need for temporal credit assignment as such”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Gershman et al., 2012
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 5×
“An important—and computationally tractable—class of such learning problems arises when the environmental dynamics are specified by a Markov chain (a key assumption in RL theory; see Sutton & Barto, 1998), and the property to be inferred has a particular functional form that we describe below.”
InfluentialReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,661
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“Here we consider a temporal difference (TD) learning algorithm (Sutton, 1988) that estimates M directly from sample paths (see White, 1995, for more information).”
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · cited 1×
“Sutton’s work served as the precursor to later theories of hierarchical RL, in which values can be defined over temporally abstract sequences of states and actions (Sutton et al., 1999).”
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
Kemeny, J. G., & Snell, J. L. (1976) · Medical Entomology and Zoology · Cited by 2,813
Research works printed but not linked: 7
- 10.1037/0278-7393.25.4.923 · doi:10.1037/0278-7393.25.4.923 · Paper page
- Mahadevan S., 2007, Journal of Machine Learning Research, 8, 2169
- 10.1037/0033-295X.95.4.434 · doi:10.1037/0033-295x.95.4.434 · Paper page
- Schank R., 1982, Dynamic memory
- Sederberg P., 2010, Psychonomic Bulletin and Review, 38, 689
- Socher R., 2009, Advances in neural information processing systems, 22, 1714
- 10.1037/0882-7974.13.2.256 · doi:10.1037/0882-7974.13.2.256 · Paper page
Cited by
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
“sentation, which represents a value function in terms of the expected future state occupancy. Decomposition of the successor representation yields reasonable sub-goals for spatial navigation problems [5, 12, 44]. Botvinick et al.[3] have written a general overview of hierarchical reinforcement learning in the context…”
BackgroundThe successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
“Using these updated representations, we compute Q(sa) at the end of phase 1 and again at the end of phase 2 using equation (5) for the SR learner and equation (6) for the MB learner.”
MethodSelf-citationPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
“In line with this, it has been noted that there is a close correspondence between the update rule used by SR-TD and update rules in the temporal context model of memory [19].”
MethodSelf-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
Self-citationRethinking dopamine as generalized prediction error
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2018) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 160
“, 2017), behavioral (Gershman et al., 2012; Momennejad et al., 2017; Smith et al., 2013) and neural (Brea et al.”
BackgroundSelf-citationLearning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
A Distributed Neural Code in the Dentate Gyrus and in CA1
Stefanini, F., Kushnir, L., Jimenez, J. C., Jennings, J. H., Woods, N. I., Stuber, G. D., Kheirbek, M. A., Hen, R., & Fusi, S. (2020) · Neuron · doi:10.1016/j.neuron.2020.05.022 · Cited by 168
“Our simulations where multiple variables are encoded are compatible with recent models of the hippocampus that emphasize its role in memory compression (Gluck and Myers, 1993; Benna and Fusi, 2019), and memory prediction (Dayan, 1993; Stachenfeld et al., 2014, 2017; Gershman et al., 2012; Recanatesi et al., 2018; Whit…”
MethodLiu, Z., Yao, C., Yu, H., & Wu, T. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.02.068 · Cited by 149
Deep Successor Reinforcement Learning
Kulkarni, T. D., Saeedi, A., Gautam, S., & Gershman, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02396 · Cited by 82
“Many learning problems involve inferring properties of temporally extended sequences given an objective function.”
BackgroundSelf-citationThe temporal dynamics of opportunity costs: A normative account of cognitive fatigue and boredom.
Agrawal, M., Mattar, M. G., Cohen, J. D., & Daw, N. D. (2022) · Psychological Review · doi:10.1037/rev0000309 · Cited by 154
A retrieved context model of the emotional modulation of memory.
Talmi, D., Lohnas, L. J., & Daw, N. D. (2019) · Psychological Review · doi:10.1037/rev0000132 · Cited by 129
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
Predictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Computational Models of Memory Search
Kahana, M. J. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-103358 · Cited by 137
Retroactive and graded prioritization of memory by reward
Braun, E. K., Wimmer, G. E., & Shohamy, D. (2018) · Nature Communications · doi:10.1038/s41467-018-07280-0 · Cited by 99
Sadacca, B. F., Wied, H. M., Lopatina, N., Saini, G. K., Nemirovsky, D., & Schoenbaum, G. (2018) · eLife · doi:10.7554/elife.30373 · Cited by 94
“These 298 representations capture the expectation of moving to one state from another, independent of 299 value, but stop short of encoding a full task model [40].”
BackgroundHayes, T. R., Petrov, A. A., & Sederberg, P. B. (2011) · Journal of Vision · doi:10.1167/11.10.10 · Cited by 103
Self-citationTransferring structural knowledge across cognitive maps in humans and models
Mark, S., Moran, R., Parr, T., Kennerley, S. W., & Behrens, T. E. J. (2020) · Nature Communications · doi:10.1038/s41467-020-18254-6 · Cited by 111
Do we really become smarter when our fluid-intelligence test scores improve?
Hayes, T. R., Petrov, A. A., & Sederberg, P. B. (2015) · Intelligence · doi:10.1016/j.intell.2014.10.005 · Cited by 93
Self-citationBenna, M. K., & Fusi, S. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2018422118 · Cited by 118
Konovalov, A., & Krajbich, I. M. (2016) · Nature Communications · doi:10.1038/ncomms12438 · Cited by 106
“The DYNA framework suggests how such learning could occur 21 and successor representation describes how this could be extended to multi-stage environments 55,56 .”
How humans learn and represent networks
Lynn, C. W., & Bassett, D. S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912328117 · Cited by 109
The Neural Representation of Prospective Choice during Spatial Planning and Decisions
Kaplan, R., King, J., Koster, R., Penny, W. D., Burgess, N., & Friston, K. J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.1002588 · Cited by 79
Design Principles of the Hippocampal Cognitive Map
Stachenfeld, K., Botvinick, M., & Gershman, S. J. (2014) · neural information processing systems · Cited by 112
Self-citationHuman information processing in complex networks
Lynn, C. W., Papadopoulos, L., Kahn, A. E., & Bassett, D. S. (2020) · Nature Physics · doi:10.1038/s41567-020-0924-7 · Cited by 86
A retrieved context account of spacing and repetition effects in free recall.
Siegel, L. L., & Kahana, M. J. (2014) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0035585 · Cited by 74
A general model of hippocampal and dorsal striatal learning and decision making
Geerts, J. P., Chersi, F., Stachenfeld, K. L., & Burgess, N. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2007981117 · Cited by 68
“This 60‐cm limit can thus make the associative‐learning strat‐ egy perform artificially poorly.”
Abstract representations of events arise from mental errors in learning and memory
Lynn, C. W., Kahn, A. E., Nyema, N., & Bassett, D. S. (2020) · Nature Communications · doi:10.1038/s41467-020-15146-7 · Cited by 82
Neural learning rules for generating flexible predictions and computing the successor representation
Fang, C., Aronov, D., Abbott, L. F., & Mackevicius, E. L. (2023) · eLife · doi:10.7554/elife.80680 · Cited by 73
“12 Inspired by work in the reinforcement learning (RL) field, these observations have been 13 formalized by describing hippocampal activity as a predictive map under the successor 14 representation (SR) algorithm [23, 24, 18].”
MethodA Predictive Processing Model of Episodic Memory and Time Perception
Fountas, Z., Sylaidi, A., Nikiforou, K., Seth, A. K., Shanahan, M., & Roseboom, W. (2022) · Neural Computation · doi:10.1162/neco_a_01514 · Cited by 77
Predicting the Future with Multi-scale Successor Representations
Momennejad, I., & Howard, M. W. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/449470 · Cited by 54
“, 2017), computational accounts of episodic memory and temporal context (Gershman et al., 2012), and predictive representations in place cells and grid cells in the hippocampus and medial entorhinal cortex (Stachenfeld et al.”
BackgroundPredicting the past, remembering the future
Gershman, S. J. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.05.025 · Cited by 58
“It can be shown (see [26] for details) that the temporal context representation corresponds to the eligibility trace et, and the matrix of item-context associations corresponds to M̂.”
BackgroundMethodSelf-citationCompound cuing in free recall.
Lohnas, L. J., & Kahana, M. J. (2014) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0033698 · Cited by 63
Temporal context and latent state inference in the hippocampal splitter signal
Duvelle, É., Grieves, R. M., & van der Meer, M. A. A. (2023) · eLife · doi:10.7554/elife.82357 · Cited by 61
Reinforcement learning with Marr
Niv, Y., & Langdon, A. (2016) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2016.04.005 · Cited by 64
“, hierarchical RL [89] and successor representation frameworks [90,91]), suggesting that diversity is perhaps a feature of the system rather than a bug.”
“Chasing the first high”: memory sampling in drug choice
Bornstein, A. M., & Pickard, H. (2020) · Neuropsychopharmacology · doi:10.1038/s41386-019-0594-2 · Cited by 55
“Selection has been shown to be influenced by recency [17], goal congruence [88], mood-congruence [85], and an incrementally-learned estimate of the contingencies between states in the environment [136].”
BackgroundPudhiyidath, A., Morton, N. W., Duran, R. V., Schapiro, A. C., Momennejad, I., Hinojosa-Rowland, D. M., Molitor, R. J., & Preston, A. R. (2022) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01864 · Cited by 56
“Such a mechanism would be consistent with temporal context models, which propose that presented items become associated with a contextual trace that represents recently presented items (Gershman et al., 2012; Howard & Kahana, 2002).”
Lu, C., Lu, H., Chen, D., Wang, H., Li, P., & Gong, J. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104328 · Cited by 43
Ho, M. K., Abel, D. L., Griffiths, T. L., & Littman, M. L. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.05.001 · Cited by 60
The context repetition effect: Predicted events are remembered better, even when they don’t happen
Smith, T. A., Hasinski, A. E., & Sederberg, P. B. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0034067 · Cited by 33
Self-citationHabit learning in hierarchical cortex–basal ganglia loops
Baladron, J., & Hamker, F. H. (2020) · European Journal of Neuroscience · doi:10.1111/ejn.14730 · Cited by 34
The learning of prospective and retrospective cognitive maps within neural circuits
Namboodiri, V. M. K., & Stuber, G. D. (2021) · Neuron · doi:10.1016/j.neuron.2021.09.034 · Cited by 41
“Essentially, the SR, expressed in a similar form as the transition matrix, measures how often the animal transitions from a given state to any other state and exponentially discounts the number of steps required for these transitions (Dayan, 1993; Gershman, 2018; Gershman et al., 2012; Momennejad et al., 2017) (exact…”
BackgroundRethinking dopamine as generalized prediction errorPreprint
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/239731 · Cited by 30
Self-citationLearning to represent reward structure: A key to adapting to complex environments
Nakahara, H., & Hikosaka, O. (2012) · Neuroscience Research · doi:10.1016/j.neures.2012.09.007 · Cited by 25
“Progress related to learning the reward strucure in representation has been ongoing in other fields apart from euroscience, such as machine learning, by using predictive states, xtracting or approximating features that represent states, or using ther types of time traces (Daw et al., 2006; Gershman et al., 2012; udvig…”
Replay shapes abstract cognitive maps for efficient social navigation
Son, J.-Y., Vives, M., Bhandari, A., & FeldmanHall, O. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01990-w · Cited by 27
“People’s use of multi-step abstraction allows them to build more holistic representations of how people in the network are connected to one another and suggests that abstraction is the linchpin of how social navigation problems are solved 3,4 .”
Between planning and map building: Prioritizing replay when future goals are uncertain
Sagiv, Y., Akam, T., Witten, I. B., & Daw, N. D. (2025) · Neuron · doi:10.1016/j.neuron.2025.09.021 · Cited by 27
Estimating Scale-Invariant Future in Continuous Time
Tiganj, Z., Gershman, S. J., Sederberg, P. B., & Howard, M. W. (2019) · Neural Computation · doi:10.1162/neco_a_01171 · Cited by 28
“…related to the temporal context model, a computational approach that has been applied to behavioral results in a range of episodic memory paradigms (TCM, Howard & Kahana, 2002; Sederberg, Howard, & Kahana, 2008; Polyn, Norman, & Kahana, 2009; Gershman, Moore, Todd, Norman, & Sederberg, 2012).”
MethodSelf-citationEpisodic retrieval for model-based evaluation in sequential decision tasks.
Zhou, C. Y., Talmi, D., Daw, N. D., & Mattar, M. G. (2025) · Psychological Review · doi:10.1037/rev0000505 · Cited by 21
“…of future investigation in a memory regime that has not seen much study in list learning: how episodic recall is affected by repeated exposure to lists with overlapping items (Gershman et al., 2012), analogous to the hypothetical transition from individual trajectories to an SR in our model.”
BackgroundMethodResultInfluentialEpisodic retrieval for model-based evaluation in sequential decision tasks
Zhou, C. Y., Talmi, D., Daw, N., & Mattar, M. G. (2023) · doi:10.31234/osf.io/3sqjh · Cited by 21
Wittkuhn, L., Krippner, L. M., Koch, C., & Schuck, N. W. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2507516122 · Cited by 29
Temporal chunking as a mechanism for unsupervised learning of task-sets
Bouchacourt, F., Palminteri, S., Koechlin, É., & Ostojic, S. (2020) · eLife · doi:10.7554/elife.50469 · Cited by 25
Predictive coding of reward in the hippocampus
Yaghoubi, M., Kumar, M. G., Nieto‐Posadas, A., Mosser, C., Gisiger, T., Wilson, É., Pehlevan, C., Williams, S., & Brandon, M. P. (2026) · Nature · doi:10.1038/s41586-025-09958-0 · Cited by 29
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15
Self-citationQuantum computing and neuroscience for 6G/7G networks: Survey
Glisic, S., & Lorenzo, B. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200346 · Cited by 19
Long-term, multi-event surprise correlates with enhanced autobiographical memory
Antony, J. W., Dam, J. V., Massey, J., Barnett, A. J., & Bennion, K. A. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01631-8 · Cited by 16
Remembering rewarding futures: A simulation‐selection model of the hippocampus
Jung, M. W., Lee, H., Jeong, Y., Lee, J. W., & Lee, I. (2018) · Hippocampus · doi:10.1002/hipo.23023 · Cited by 18
“value-processing brain structures such as the ventral striatum (Gershman et al., 2012; Johnson, van der Meer, & Redish, 2007; Pezzulo et al., 2014; van der Meer, Kurth-Nelson, & Redish, 2012; Yu & Frank, 2015).”
BackgroundInformation content of note transitions in the music of J. S. Bach
Kulkarni, S., David, S. U., Lynn, C. W., & Bassett, D. S. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.013136 · Cited by 21
“We emphasize the similarity of this form across multiple different theories of cognition [30, 33, 34].”
BackgroundSynaptic encoding of temporal contiguity
Ostojic, S., & Fusi, S. (2013) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00032 · Cited by 15
“…this an important step toward the understanding of the biological substrate of the successor representation in reinforcement learning (Dayan, 1993; Gershman et al, 2012), and more generally for Markov Decision Processes (Bellman, 1957), which play an important role for decision making in complex…”
BackgroundRigid reduced successor representation as a potential mechanism for addiction
Shimomura, K., Kato, A., & Morita, K. (2021) · European Journal of Neuroscience · doi:10.1111/ejn.15227 · Cited by 18
Cunningham, P. J., Regier, P. S., & Redish, A. D. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.3080-20.2021 · Cited by 18
Kahana, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-809324-5.21038-9 · Cited by 18
“…variability in memory encoding is not simply a result of stochastic variation in attention or learning parameters, but that the memory system uses prediction to control the learning of new information and the way in which memories are organized into specific “events” (e.g., Gershman et al., 2012).”
BackgroundA distributed neural code in the dentate gyrus and in CA1Preprint
Stefanini, F., Kheirbek, M. A., Kushnir, L., Jimenez, J., Jennings, J. H., Stuber, G. D., Hen, R., & Fusi, S. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/292953 · Cited by 16
Hierarchical Intermittent Motor Control With Deterministic Policy Gradient
Shi, H., Sun, Y., Li, G., Wang, F., Wang, D., & Li, J. (2019) · IEEE Access · doi:10.1109/access.2019.2904910 · Cited by 14
Mice identify subgoal locations through an action-driven mapping process
Shamash, P., Lee, S., Saxe, A., & Branco, T. (2023) · Neuron · doi:10.1016/j.neuron.2023.03.034 · Cited by 18
What to Choose Next? A Paradigm for Testing Human Sequential Decision Making
Tartaglia, E. M., Clarke, A., & Herzog, M. H. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00312 · Cited by 18
“For this experiment we also tried to fit the data to the Successor representation (SR) model (Dayan, 1993; Gershman et al., 2012; Momennejad et al., 2016) and to a hybrid mixture (HM) ofmodel-free andmodel- based algorithms (Gläscher et al.”
MethodThe influence of emotion on temporal context models
Lohnas, L. J., & Howard, M. W. (2024) · Cognition & Emotion · doi:10.1080/02699931.2024.2371075 · Cited by 17
“Classical TCMs have provided a theoretical framework for the formation of these predictions, informed by memories of the temporal history of experiences (Bornstein & Norman, 2017; Franklin et al., 2020; Gershman et al., 2012).”
Memory and Planning in Brains and Machines: Multiscale Predictive Representations
Momennejad, I. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192882547.003.0012 · Cited by 14
Toward the Emergence of Intelligent Control: Episodic Generalization and Optimization
Giallanza, T., Campbell, D. I., & Cohen, J. D. (2024) · Open Mind · doi:10.1162/opmi_a_00143 · Cited by 22
“…explained in terms of combination of model-based decision making (Daw et al., 2011) and the learning of successor representations (Dayan, 1993; Gershman et al., 2012) in prediction and decision-making; 2) the effects of blocked versus interleaved training in Beukers, Collin, et al. ’ s (2024)…”
BackgroundNeurophysiological Evidence for Cognitive Map Formation during Sequence Learning
Stiso, J., Lynn, C. W., Kahn, A. E., Rangarajan, V., Szymula, K. P., Archer, R., Revell, A. Y., Stein, J. M., Litt, B., Davis, K. A., et al. (2022) · eNeuro · doi:10.1523/eneuro.0361-21.2022 · Cited by 13
“Computational models of behavior that require learning an underlying latent space bear a striking resemblance to those used for learning and navigating Euclidean or abstract relational spaces[12, 13].”
BackgroundDual Reward Prediction Components Yield Pavlovian Sign- and Goal-Tracking
Kaveri, S., & Nakahara, H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0108142 · Cited by 13
“In terms of computational techniques, the predictive state [45,46] or successor [47,48] representations may be applied.”
BackgroundCase-Based Decision Neuroscience
Bhui, R. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812098-9.00004-8 · Cited by 13
“Intriguingly, the successor representation has been linked to the temporal context model of episodic memory by Gershman et al. (2012).”
BackgroundThe neural architecture of prediction over a continuum of spatiotemporal scales
Gravina, M. T., & Sederberg, P. B. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.09.001 · Cited by 10
Self-citationManning, J. R. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190917982.013.38 · Cited by 10
False memories for ending of events.
Raykov, P., Varga, D., & Bird, C. M. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001462 · Cited by 17
“Exactly how our predictions are violated, by omission or commission of information, is likely to have a different effect on learning (Gershman et al., 2012; Kim et al., 2014, 2017; T. A. Smith et al., 2013).”
Feng, Z., Nagase, A. M., & Morita, K. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.660595 · Cited by 11
Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus
Mehrotra, D., & Dubé, L. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1200842 · Cited by 11
“SR models can provide a computational framework for episodic learning (Gershman et al., 2012), and have demonstrated that the representation of space and perhaps other variables of interest is not merely a static representation, but a predictive one, encoding the statistics of future expectations…”
Proactive and reactive construction of memory-based preferences
Nicholas, J., Daw, N. D., & Shohamy, D. (2025) · Nature Communications · doi:10.1038/s41467-025-56183-4 · Cited by 9
Compositional Sequence Generation in the Entorhinal–Hippocampal System
McNamee, D., Stachenfeld, K., Botvinick, M., & Gershman, S. J. (2022) · Entropy · doi:10.3390/e24121791 · Cited by 11
“For example, an animal may seek to maintain an internal representation of its present context over a longer timescale compared to encoding the detailed sensory representation of its current position during a traversal of an environment [47,48].”
BackgroundSelf-citationLearning Temporal Relationships Between Symbols with Laplace Neural Manifolds
Howard, M. W., Esfahani, Z. G., Le, B., & Sederberg, P. B. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00230-8 · Cited by 13
“As such it provides a neurally-reasonable mechanism to estimate temporal contingency (Balsam & Gallistel, 2009; Gallistel, 2021b; Jeong et al., 2022).”
Self-citationSuccessor Feature Sets: Generalizing Successor Representations Across Policies
Brantley, K., Mehri, S., & Gordon, G. J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17399 · Cited by 12
Memory Reconsolidation: Making Predictions Better
Nadel, L., & Sederberg, P. B. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190917982.013.44 · Cited by 10
Self-citationTarder-Stoll, H., Baldassano, C. A., & Aly, M. (2024) · Psychological Science · doi:10.1177/09567976241256617 · Cited by 8
“This model assumed that increasingly distant future states would be increasingly difficult to access (consistent with temporal discounting; Gershman et al., 2012) but that the degree of difficulty could be different for different cues.”
Cognitive graphs: Representational substrates for planning.
Yoo, J., Chrastil, E. R., & Bornstein, A. M. (2024) · Decision · doi:10.1037/dec0000249 · Cited by 10
“Backed-up representations enable fast, cheap evaluation of N-step plan outcomes, using an operation akin to matrix multiplication (though the neural instantiation of this process has yet to be fully described (Gershman, 2018) and may be approximated by sampling (Gershman et al., 2012)).”
AKF-SR: Adaptive Kalman filtering-based successor representation
Malekzadeh, P., Salimibeni, M., Hou, M., Mohammadi, A., & Plataniotis, K. N. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.008 · Cited by 9
“(3), the SR can be also updated in an on-policy recursive form [30], known as SARSA, as follows”
BackgroundResultInfluentialDarby, K., & Sederberg, P. B. (2022) · Psychology and Aging · doi:10.1037/pag0000665 · Cited by 11
Self-citationThe detour problem in a stochastic environment: Tolman revisited
Fakhari, P., Khodadadi, A., & Busemeyer, J. R. (2017) · Cognitive Psychology · doi:10.1016/j.cogpsych.2017.12.002 · Cited by 7
“There is a third RL algorithm, called successor representation, that learns a rough representation of the environment by storing the expected future visits of each state, Gershman et al. (2012), Dayan (1993). It is computationally less expensive than the model-based RL and can easily explain the planning behavior in o…”
MethodInfluentialTrial-by-trial learning of successor representations in human behavior
Kahn, A. E., Bassett, D. S., & Daw, N. D. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013696 · Cited by 11
“Since the first article introducing the SR (Dayan 1993), it has been widely assumed that the SR is directly learned via temporal-difference learning (“SR-TD”), specifically TD(0) bootstrapping (Dayan 1993; Gershman et al. 2012; Gardner, Schoenbaum, and Gershman 2018).”
Lu, H., Lu, C., Wang, H., Gong, J., Zhu, M., & Yang, H. (2024) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2024.104899 · Cited by 8
Malekzadeh, P., Hou, M., & Plataniotis, K. N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.01.076 · Cited by 8
“Gershman et al. [123] indicated that, like the Q-value function in MF algorithms, the SR could be directly updated in a recursive form without the use of a one-step transition model T π , as follows: In this paper, the methods that use Eq.”
MethodReconciling time and prediction error theories of associative learning
Hamou, N., Gershman, S. J., & Reddy, G. (2025) · Nature Communications · doi:10.1038/s41467-025-65137-9 · Cited by 6
“…to highlight the in fl uence of reward pre-conditioning, propose a noisy accumulator model to explain timescale invariance in response pro fi les 28 , and predictive representations of stimulus-reward intervals 12,26,31,32 to explain how an animal could form associations across multiple timescales.”
Self-citationComputational Models of Episodic Memory
Sederberg, P. B., & Darby, K. (2023) · Cambridge University Press eBooks · doi:10.1017/9781108755610.022 · Cited by 4
Self-citationSAMPL: The Spreading Activation and Memory PLasticity ModelPreprint
Sievers, B., & Momennejad, I. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/778563 · Cited by 3
“Furthermore, it has been shown that the temporal context model (TCM) 15 is a special case of the successor representation, offering simple update principles for organization of and generalization across memory networks 5 .”
BackgroundMethodInfluentialThe construction and use of cognitive maps in model-based controlPreprint
Karagoz, A. B., Reagh, Z. M., & Kool, W. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ngqwa · Cited by 3
Duncan, K. D., & Shohamy, D. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190917982.013.24 · Cited by 5
Towards a unifying theory of generalizationPreprint
Schulz, E. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/rzj2m · Cited by 2
“…that the successor representation of reinforcement learning (Dayan, 1993) has recently been shown to play a role in learning, memory and control (Gershman et al., 2012; Momennejad et al., 2017) and that it is possible to define a kernel using the inverse graph Laplacian (the successor…”
BackgroundFujimoto, S., Meger, D., & Precup, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06854 · Cited by 2
“It has also been used in a neuroscience context to model generalization and human reinforcement learning (Gershman et al., 2012; Momennejad et al., 2017; Gershman, 2018).”
MethodLong short-term prediction guides human metacognitive reinforcement learningPreprint
Lee, S. W., Kim, D., Lee, J. H., Jung, W., & Kim, S. H. (2023) · Research Square · doi:10.21203/rs.3.rs-3080402/v1 · Cited by 2
Statistical learning drives predictive shifts in object memoryPreprint
Singh, D., Dong, C., Tandoc, M. C., & Schapiro, A. C. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/yuxb6_v1 · Cited by 4
Efficient Learning of Predictive Maps for Flexible PlanningPreprint
Bazarjani, A., & Piray, P. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.11.705395 · Cited by 3
“These mazes are ideal for demonstrating this issue because the starting locations alternate between opposite sides—left/right in Maze 15 and top/bottom in Maze 22.”
Metadata
- Authors
- Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B.
- Affiliations
- Princeton University (US)
- The Ohio State University (US)
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2012-06
- Volume
- 24
- Issue
- 6
- Pages
- 1553-1568
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2076546217
- Cited by
- 117
- References
- 61
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 12 · 2021: 10 · 2020: 11 · 2019: 7 · 2018: 8 · 2017: 12 · 2016: 6 · 2014: 4 · 2013: 3 · 2012: 1
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- successor representation, temporal difference learning, episodic memory, episodic learning, reinforcement learning, normative model, memory generalization, learning systems
- MeSH terms
- Algorithms, Learning, Models, Psychological, Time Factors, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.