A probabilistic successor representation for context-dependent learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 61 of the 83 references it cites were found in a registry or an index, and 22 are not checked yet.
Abstract
Two of the main impediments to learning complex tasks are that relationships between different stimuli, including rewards, can be uncertain and context-dependent. Reinforcement learning (RL) provides a framework for learning, by predicting total future reward directly (model-free RL), or via predictions of future states (model-based RL). Within this framework, "successor representation" (SR) predicts total future occupancy of all states. A recent theoretical proposal suggests that the hippocampus encodes the SR in order to facilitate prediction of future reward. However, this proposal does not take into account how learning should adapt under uncertainty and switches of context. Here, we introduce a theory of learning SRs using prediction errors which includes optimally balancing uncertainty in new observations versus existing knowledge. We then generalize that approach to a multicontext setting, allowing the model to learn and maintain multiple task-specific SRs and infer which one to use at any moment based on the accuracy of its predictions. Thus, the context used for predictions can be determined by both the contents of the states themselves and the distribution of transitions between them. This probabilistic SR model captures animal behavior in tasks which require contextual memory and generalization, and unifies previous SR theory with hippocampal-dependent contextual decision-making. (PsycInfo Database Record (c) 2024 APA, all rights reserved).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“rather than encoding a single monolithic predictive map of the environment, the hippocampus encodes multiple separate predictive maps, each associated with its own context.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Both the context and the predictive map are inferred using Bayesian inference”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“A new context is inferred when the current predictive map fails to account for current observations.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“learning of the predictive map is modulated by uncertainty, and supports non-local updating.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A New Approach to Linear Filtering and Prediction Problems
Kalman, R. E. (1960) · Journal of Basic Engineering · doi:10.1115/1.3662552 · Cited by 31,751
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 2×
“TD learning of the value function (using reward rather than state prediction errors) is a popular model of learning in the striatum (Schultz et al., 1997), and TD algorithms can be implemented in biologically realistic spiking networks (Bono et al.”
BackgroundThe hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 1×
“…a rich body of work in psychology and neuroscience suggesting that multiple learning systems operate in parallel, competing for control of behavior (Daw et al., 2005; Geerts et al., 2020) and that there are alternative approaches that combine aspects of model-based and model-free learning…”
BackgroundThe hippocampal-VTA loop: controlling the entry of information into long-term memory
Lisman, J. E., & Grace, A. A. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.002 · Cited by 1,983
Context and Behavioral Processes in Extinction: Table 1.
Bouton, M. E. (2004) · Learning & Memory · doi:10.1101/lm.78804 · Cited by 1,808 · cited 1×
“There is plentiful evidence that animals can deal flexibly with change, and understanding how context modulates behavior is a longstanding topic of interest in psychology and neuroscience (Bouton, 2004; Bouton & Bolles, 1979; Gershman, 2017a; Gershman et al., 2010; Heald et al., 2021).”
BackgroundKhalifa, A. A. Z., Ichikawa, M., Dai, D., Kubo, S., Black, C. S., Peri, K., McAlear, T. S., Veyron, S., Yang, S. K., Vargas, J., et al. (2020) · eLife · doi:10.7554/elife.52760 · Cited by 1,676
Bayesian Reasoning and Machine Learning
Barber, D. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804779 · Cited by 1,631
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 1×
“This is different from experience replay, in which experienced sequences of states are replayed (Lin, 1992), and simulated experience, in which possible experienced sequences are generated from a transition model (Momennejad, Otto, et al., 2017; Sutton, 1991); rather, multiple states are…”
BackgroundPredictability, surprise, attention, and conditioning
Kamin, L. J. (1967) · NASA Technical Reports Server (NASA) · Cited by 1,483
Churchill, A. J., Gutiérrez, G. D., Singer, R. A., Lorberbaum, D. S., Fischer, K. A., & Sussel, L. (2017) · eLife · doi:10.7554/elife.20010 · Cited by 1,236
Encoding Predictive Reward Value in Human Amygdala and Orbitofrontal Cortex
Gottfried, J. A., O'Doherty, J., & Dolan, R. J. (2003) · Science · doi:10.1126/science.1087919 · Cited by 1,165 · cited 1×
“As another possibility, the OFC has previously been shown to be involved in predicting both reward outcomes (Gottfried et al., 2003; Schoenbaum et al., 1998) and sensory events (Chaumon et al., 2014) and is crucial for learning the stimulus–stimulus associations in Hart et al. (2020; Figure 5).”
BackgroundThe hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationOrbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · cited 1×
“Accordingly, Wilson et al. (2014) have proposed that the OFC encodes a cognitive map of task space.”
BackgroundOrbitofrontal cortex and basolateral amygdala encode expected outcomes during learning
Schoenbaum, G., Chiba, A. A., & Gallagher, M. (1998) · Nature Neuroscience · doi:10.1038/407 · Cited by 850 · cited 1×
“As another possibility, the OFC has previously been shown to be involved in predicting both reward outcomes (Gottfried et al., 2003; Schoenbaum et al., 1998) and sensory events (Chaumon et al., 2014) and is crucial for learning the stimulus–stimulus associations in Hart et al. (2020; Figure 5).”
BackgroundDyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1×
“…1992), and simulated experience, in which possible experienced sequences are generated from a transition model (Momennejad, Otto, et al., 2017; Sutton, 1991); rather, multiple states are potentially activated and updated simultaneously (like the form of reactivation described in Manning,…”
BackgroundImproving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693 · cited 5×
“This successor representation (SR; Dayan, 1993) combines the aspects of model-based and model-free RL. Unlike model-based methods, reward prediction with the SR can be done without explicit forward simulation and is therefore relatively computationally efficient.”
BackgroundMethodInfluentialObserved increasing water constraint on vegetation growth over the last three decades
Jiao, W., Wang, L., Smith, W. K., Chang, Q., Wang, H., & D’Odorico, P. (2021) · Nature Communications · doi:10.1038/s41467-021-24016-9 · Cited by 653
Role of conditioned contextual stimuli in reinstatement of extinguished fear.
Bouton, M. E., & Bolles, R. C. (1979) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037//0097-7403.5.4.368 · Cited by 530 · cited 1×
“There is plentiful evidence that animals can deal flexibly with change, and understanding how context modulates behavior is a longstanding topic of interest in psychology and neuroscience (Bouton, 2004; Bouton & Bolles, 1979; Gershman, 2017a; Gershman et al., 2010; Heald et al., 2021).”
BackgroundExperience-Dependent Asymmetric Shape of Hippocampal Receptive Fields
Mehta, M. R., Quirk, M. C., & Wilson, M. A. (2000) · Neuron · doi:10.1016/s0896-6273(00)81072-7 · Cited by 481 · cited 1×
“For example, the firing fields of hippocampal place cells show experience-dependent skewing, consistent with a prediction of future locations (Mehta et al., 2000).”
BackgroundLong-term plasticity in hippocampal place-cell representation of environmental geometry
Lever, C., Wills, T., Cacucci, F., Burgess, N., & O'Keefe, J. (2002) · Nature · doi:10.1038/416090a · Cited by 465 · cited 1×
“In Lever et al. (2002), maps for environments of different geometries differentiate within trials but they get more similar again between trials.”
BackgroundSelf-citationBayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Self-citationContext, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 3×
“To model the multiple context setting, we build on the literature that casts context as a latent state that the animal needs to infer from observations (Gershman et al., 2010; Sanders et al., 2020).”
BackgroundMethodSelf-citationHippocampus and context in classical conditioning
Holland, P. C., & Bouton, M. E. (1999) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(99)80027-0 · Cited by 383
A map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Dearden, R., Friedman, N., & Russell, S. (1998) · Cited by 363
Reinforcement learning with Gaussian processes
Engel, Y., Mannor, S., & Meir, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102377 · Cited by 335
Predictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326 · cited 16×
“, 2020) and that there are alternative approaches that combine aspects of model-based and model-free learning (Gershman, 2018; Russek et al., 2017).”
BackgroundInfluentialSelf-citationDopamine transients are sufficient and necessary for acquisition of model-based associations
Sharpe, M. J., Chang, C. Y., Liu, M. A., Batchelor, H. M., Mueller, L. E., Jones, J. L., Niv, Y., & Schoenbaum, G. (2017) · Nature Neuroscience · doi:10.1038/nn.4538 · Cited by 300 · cited 1×
“As for the prediction error used to update the SR, evidence from optogenetic studies in rodents suggests that these could be encoded by a population of dopamine (DA) neurons in the ventral tegmental area (see Figure A1; Gardner et al., 2018; Sharpe et al., 2017).”
MethodSchapiro, A. C., Turk-Browne, N. B., Norman, K. A., & Botvinick, M. M. (2016) · Hippocampus · doi:10.1002/hipo.22523 · Cited by 294 · cited 1×
“Neuroimaging studies have furthermore shown predictive coding of nonspatial states (Garvert et al., 2017; Schapiro et al., 2016).”
BackgroundPutting Fear in Its Place: Remapping of Hippocampal Place Cells during Fear Conditioning
Moita, M. A. P., Rosis, S., Zhou, Y., LeDoux, J. E., & Blair, H. T. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.5492-03.2004 · Cited by 251
Contextual inference underlies the learning of sensorimotor repertoires
Heald, J. B., Lengyel, M., & Wolpert, D. M. (2021) · Nature · doi:10.1038/s41586-021-04129-3 · Cited by 238 · cited 1×
“There is plentiful evidence that animals can deal flexibly with change, and understanding how context modulates behavior is a longstanding topic of interest in psychology and neuroscience (Bouton, 2004; Bouton & Bolles, 1979; Gershman, 2017a; Gershman et al., 2010; Heald et al., 2021).”
BackgroundBayesian Nonparametric Inference of Switching Dynamic Linear Models
Fox, E., Sudderth, E. B., Jordan, M. I., & Willsky, A. S. (2011) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2010.2102756 · Cited by 229 · cited 5×
“After choosing a context, the generative model proceeds by evolving the state variable for each previously active context k according to the evolution equation of the LDS:mkj,t ∼N ðmkj,t−1,QÞ.”
MethodInfluentialFrom contextual fear to a dynamic view of memory systems
Fanselow, M. S. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.10.008 · Cited by 227 · cited 1×
“It has been shown that a stronger conditioned response is evoked if animals are able to explore the new environment for several minutes before the first shock, a finding known as the “context preexposure facilitation effect” (Fanselow, 2010).”
BackgroundOn-Line Inference for Hidden Markov Models via Particle Filters
Fearnhead, P., & Clifford, P. (2003) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00421 · Cited by 221 · cited 1×
“Note that the collapsing step is similar to the resampling step in particle filtering (Fearnhead & Clifford, 2003).”
MethodReinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons
Frémaux, N., Sprekeler, H., & Gerstner, W. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003024 · Cited by 209
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · cited 3×
“…and neuroscience suggesting that multiple learning systems operate in parallel, competing for control of behavior (Daw et al., 2005; Geerts et al., 2020) and that there are alternative approaches that combine aspects of model-based and model-free learning (Gershman, 2018; Russek et al., 2017).”
BackgroundSelf-citationA Unifying Probabilistic View of Associative Learning
Gershman, S. J. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004567 · Cited by 166 · cited 8×
“…can infer information about the hidden SR weights using Bayes’ rule: pðmj,tjϕ1∶tÞ ∝ pðϕ1∶tjmj,tÞpðmj,tÞ: (8) This idea, in the form of a Kalman filter (Kalman, 1960), has previously been applied to learning value functions (Geist & Pietquin, 2010; Gershman, 2015) and readily applies to the SR.”
BackgroundMethodInfluentialSelf-citationHippocampal remapping as hidden state inference
Sanders, H., Wilson, M. A., & Gershman, S. J. (2020) · eLife · doi:10.7554/elife.51140 · Cited by 163
Self-citationRethinking dopamine as generalized prediction error
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2018) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 160 · cited 6×
“This was also observed by Gardner et al. (2018), who simulated a very similar task with an SR model endowed with the ability to simulate offline experience (see Appendix A, Figure A1).”
BackgroundMethodInfluentialSelf-citationLearning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
A model for learning based on the joint estimation of stochasticity and volatility
Piray, P., & Daw, N. D. (2021) · Nature Communications · doi:10.1038/s41467-021-26731-9 · Cited by 153 · cited 2×
“…phenomena such as backward blocking (Dayan & Yu, 2003; Gershman, 2015) and more sophisticated Kalman filter models in which stochasticity and volatility parameters are estimated from data can further extend explanations to effects like the robustness of partial reinforcement (Piray & Daw, 2021a).”
BackgroundExploring a latent cause theory of classical conditioning
Gershman, S. J., & Niv, Y. (2012) · Learning & Behavior · doi:10.3758/s13420-012-0080-8 · Cited by 149 · cited 1×
“Similarly, for models that, like the model presented here, infer discrete latent causes, the hypothesis that the conditioning and extinction phases are generated by the same latent cause is more likely in the partial reinforcement condition (Gershman & Niv, 2012).”
Self-citationPartial reinforcement in autoshaping with pigeons
Gibbon, J., Farrell, L., Locurto, C., Duncan, H. J., & Terrace, H. S. (1980) · Animal Learning & Behavior · doi:10.3758/bf03209729 · Cited by 147 · cited 1×
“The opposite is true; however, extinction is slower after partial reinforcement, a phenomenon known as the partial reinforcement extinction effect (PREE; Gibbon et al., 1980; Haselgrove et al., 2004; Rescorla, 1999).”
BackgroundKiernan, M. J., & Westbrook, R. F. (1993) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640749308401089 · Cited by 132 · cited 3×
“Kiernan and Westbrook (1993) showed that there is an intriguing time course to these phenomena: A brief amount of preexposure facilitates subsequent learning, whereas extended preexposure to the environment inhibits further associative learning (Figure 3A).”
BackgroundMethodWinocur, G., Frankland, P. W., Sekeres, M., Fogel, S., & Moscovitch, M. (2009) · Learning & Memory · doi:10.1101/lm.1447209 · Cited by 120 · cited 3×
“Upon preexposure to Context A, when Figure 6 Contextual Memory Experiment by Winocur et al. (2009) Note.”
BackgroundMethodInfluentialPredictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Visual Predictions in the Orbitofrontal Cortex Rely on Associative Content
Chaumon, M., Kveraga, K., Barrett, L. F., & Bar, M. (2014) · Cerebral Cortex · doi:10.1093/cercor/bht146 · Cited by 108 · cited 1×
“As another possibility, the OFC has previously been shown to be involved in predicting both reward outcomes (Gottfried et al., 2003; Schoenbaum et al., 1998) and sensory events (Chaumon et al., 2014) and is crucial for learning the stimulus–stimulus associations in Hart et al. (2020; Figure 5).”
BackgroundDayan, P., & Jyu, A. (2003) · IETE Journal of Research · doi:10.1080/03772063.2003.11416335 · Cited by 105 · cited 2×
“For example, a wide range of animal learning phenomena can be explained by probabilistic generalizations of simple model-free learning algorithms, such as the Kalman filter (Dayan &Kakade, 2001; Dayan & Yu, 2003; Gershman, 2015).”
BackgroundMethodPredictive Representations in Hippocampal and Prefrontal Hierarchies
Brunec, I. K., & Momennejad, I. (2022) · The Journal of Neuroscience · doi:10.1523/jneurosci.1327-21.2021 · Cited by 105
Statistical computations underlying the dynamics of memory updating
Gershman, S. J., Radulescu, A., Norman, K. A., & Niv, Y. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003939 · Cited by 101 · cited 4×
“Next, we describe how to generalize this for multiple environments and context-dependent SR maps by using a nonparametric switching linear dynamical system (Fox et al., 2011; Gershman et al., 2014; Murphy, 1998) that infers new maps when observations change drastically over time.”
MethodInfluentialSelf-citationSchuette, P. J., Reis, F. M., Maesta‐Pereira, S., Chakerian, M., Torossian, A., Blair, G. J., Wang, W., Blair, H. T., Fanselow, M. S., Kao, J. C., et al. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.1022-20.2020 · Cited by 85
Geist, M., & Pietquin, O. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3077 · Cited by 82 · cited 3×
“Second, following the value estimation method described by Geist and Pietquin (2010), we chose a random walk model for describing the evolution process on the SR parameters.”
MethodFlexible modulation of sequence generation in the entorhinal–hippocampal system
McNamee, D., Stachenfeld, K., Botvinick, M., & Gershman, S. J. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00831-7 · Cited by 82 · cited 1×
“Replay and exploration both depend on an ability to generate sequential samples and correspond to different optimal sampling regimes, possibly mediated by entorhinal grid cells (McNamee et al., 2021).”
BackgroundSelf-citationHaselgrove, M., Aydin, A., & Pearce, J. M. (2004) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037/0097-7403.30.3.240 · Cited by 81 · cited 1×
“The opposite is true; however, extinction is slower after partial reinforcement, a phenomenon known as the partial reinforcement extinction effect (PREE; Gibbon et al., 1980; Haselgrove et al., 2004; Rescorla, 1999).”
BackgroundLinear reinforcement learning in planning, grid fields, and cognitive control
Piray, P., & Daw, N. D. (2021) · Nature Communications · doi:10.1038/s41467-021-25123-3 · Cited by 78 · cited 2×
“…phenomena such as backward blocking (Dayan & Yu, 2003; Gershman, 2015) and more sophisticated Kalman filter models in which stochasticity and volatility parameters are estimated from data can further extend explanations to effects like the robustness of partial reinforcement (Piray & Daw, 2021a).”
BackgroundDaw, N. D., & Courville, A. C. (2007) · neural information processing systems · Cited by 77
Forward and Backward Inference in Spatial Cognition
Penny, W. D., Zeidman, P., & Burgess, N. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003383 · Cited by 68
Self-citationA general model of hippocampal and dorsal striatal learning and decision making
Geerts, J. P., Chersi, F., Stachenfeld, K. L., & Burgess, N. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2007981117 · Cited by 68 · cited 1×
“…in psychology and neuroscience suggesting that multiple learning systems operate in parallel, competing for control of behavior (Daw et al., 2005; Geerts et al., 2020) and that there are alternative approaches that combine aspects of model-based and model-free learning (Gershman, 2018; Russek et…”
BackgroundSelf-citationContext Learning in the Rodent Hippocampus
Fuhs, M. C., & Touretzky, D. S. (2007) · Neural Computation · doi:10.1162/neco.2007.19.12.3173 · Cited by 62
Enhanced discriminability and reduced associability following flavor preexposure
Honey, R. C., & Hall, G. (1989) · Learning and Motivation · doi:10.1016/0023-9690(89)90008-8 · Cited by 61 · cited 1×
“In the argument of Honey and Hall (1989), increased discrimination between contexts can arise directly as a result of latent inhibition.”
BackgroundMulti-task reinforcement learning in humans
Tomov, M. S., Schulz, E., & Gershman, S. J. (2021) · Nature Human Behaviour · doi:10.1038/s41562-020-01035-y · Cited by 59 · cited 1×
“…= E X∞ t=0 γtIðst = s′Þjs0 = s : (3) Factorizing value into an SR term and a reward term permits greater flexibility because if one term changes, it can be relearned, while the other remains intact (Barreto et al., 2016; Dayan, 1993; Gershman, 2018; Russek et al., 2017; Tomov et al., 2021).”
BackgroundSelf-citationHart, E. E., Sharpe, M. J., Gardner, M., & Schoenbaum, G. M. (2020) · eLife · doi:10.7554/elife.59998 · Cited by 59 · cited 2×
“Like the animals in Hart et al. (2020) and the replay-endowed model, Kalman SR was sensitive to the reward devaluation paradigm (Figure 5D).”
MethodResultExtinction from a rationalist perspective
Gallistel, C. R. (2012) · Behavioural Processes · doi:10.1016/j.beproc.2012.02.008 · Cited by 57
WITHIN-SUBJECT PARTIAL REINFORCEMENT EXTINCTION EFFECT IN AUTOSHAPING
Rescorla, R. A. (2010) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/713932693 · Cited by 56 · cited 1×
“The opposite is true; however, extinction is slower after partial reinforcement, a phenomenon known as the partial reinforcement extinction effect (PREE; Gibbon et al., 1980; Haselgrove et al., 2004; Rescorla, 1999).”
BackgroundNeurobiological successor features for spatial navigation
de Cothi, W., & Barry, C. (2020) · Hippocampus · doi:10.1002/hipo.23246 · Cited by 53 · cited 1×
“The SR has been proposed to explain the aspects of neural data in the rodent and human hippocampus (Brunec & Momennejad, 2022; de Cothi & Barry, 2020; Gershman, 2018; Momennejad, Otto, et al., 2017; Russek et al., 2017, 2021; Stachenfeld et al., 2017) and the aspects of human and rodent behavior…”
MethodLearning predictive cognitive maps with spiking neurons during behavior and replays
Bono, J., Zannone, S., Pedrosa, V., & Clopath, C. (2023) · eLife · doi:10.7554/elife.80671 · Cited by 48 · cited 2×
“TD learning of the value function (using reward rather than state prediction errors) is a popular model of learning in the striatum (Schultz et al., 1997), and TD algorithms can be implemented in biologically realistic spiking networks (Bono et al., 2021; Brea et al., 2016; Frémaux et al., 2013).”
BackgroundHippocampal place cells encode global location but not connectivity in a complex space
Duvelle, É., Grieves, R. M., Liu, A., Jedidi-Ayoub, S., Holeniewska, J., Harris, A., Nyberg, N., Donnarumma, F., Lefort, J. M., Jeffery, K. J., et al. (2021) · Current Biology · doi:10.1016/j.cub.2021.01.005 · Cited by 48 · cited 1×
“It should be noted, however, that a recent direct neuroimaging test of the SR proved inconclusive about the role of the hippocampus (Russek et al., 2021) and that a study in rodents did not find evidence for SR coding in dorsal CA1 (Duvelle et al., 2021).”
ResultNeural evidence for the successor representation in choice evaluationPreprint
Russek, E. M., Momennejad, I., Botvinick, M., Gershman, S. J., & Daw, N. D. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.08.29.458114 · Cited by 42 · cited 2×
“…neural data in the rodent and human hippocampus (Brunec & Momennejad, 2022; de Cothi & Barry, 2020; Gershman, 2018; Momennejad, Otto, et al., 2017; Russek et al., 2017, 2021; Stachenfeld et al., 2017) and the aspects of human and rodent behavior (De Cothi et al., 2022; Momennejad, Russek, et al.,…”
BackgroundResultSelf-citationProspective Coding by Spiking Neurons
Brea, J., Gaál, A. T., Urbanczik, R., & Senn, W. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005003 · Cited by 40 · cited 1×
“TD learning of the value function (using reward rather than state prediction errors) is a popular model of learning in the striatum (Schultz et al., 1997), and TD algorithms can be implemented in biologically realistic spiking networks (Bono et al., 2021; Brea et al., 2016; Frémaux et al., 2013).”
BackgroundDopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · cited 3×
“Applying a prediction error update to this “replay vector” approximates the Kalman TD algorithm (Gershman, 2017b).”
BackgroundMethodSelf-citationChang, S., & Liang, K. (2016) · Hippocampus · doi:10.1002/hipo.22679 · Cited by 39 · cited 1×
“This does not have to be the same environment as long as the animal infers it to be the same (Chang & Liang, 2017): For instance, when uncertainty is high, the animal might falsely infer Jesse P. Geerts https://orcid.org/0000-0003-2960-5727 The authors thank Athena Akrami, Peter Dayan, and Steven…”
BackgroundBetter transfer learning with inferred successor mapsPreprint
Madarász, T. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07663 · Cited by 32
Episodic memory: Mental time travel or a quantum “memory wave” function?
Manning, J. R. (2021) · Psychological Review · doi:10.1037/rev0000283 · Cited by 30
Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning
Janz, D., Hron, J., Mazur, P., Hofmann, K., Hernández-Lobato, J. M., & Tschiatschek, S. (2018) · arXiv · doi:10.48550/arxiv.1810.06530 · Cited by 23
Probabilistic Successor Representations with Kalman Temporal Differences
Geerts, J., Stachenfeld, K., & Burgess, N. (2019) · Conference on Cognitive Computational Neuroscience (CCN) · doi:10.32470/ccn.2019.1323-0 · Cited by 17 · cited 1×
“Our first contribution, which we initially described in Geerts et al. (2019), is to introduce a probabilistic SR, in which the agent’s belief about the parameters of the SR is expressed in terms of a distribution over possible SRs.”
BackgroundSelf-citationCoordinated hippocampal-entorhinal replay as structural inference
Evans, T., & Burgess, N. (2019) · UCL Discovery (University College London) · Cited by 15
Self-citationLatent learning drives sleep-dependent plasticity in distinct CA1 subpopulationsPreprint
Guo, W., Zhang, J. J., Newman, J. P., & Wilson, M. A. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.02.27.967794 · Cited by 14 · cited 1×
“Guo et al. (2020) found evidence that a single coherent map is being built during sleep.”
BackgroundAKF-SR: Adaptive Kalman filtering-based successor representation
Malekzadeh, P., Salimibeni, M., Hou, M., Mohammadi, A., & Plataniotis, K. N. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.008 · Cited by 9 · cited 1×
“In addition to prioritizing sequences of replayed states, information about uncertainty may also be used to direct exploration to states with high uncertainty (see Malekzadeh et al., 2022, for an application of Kalman SR in active learning).”
MethodOffline Replay Supports Planning: fMRI Evidence from Reward Revaluation
Momennejad, I., Ross Otto, A., Daw, N. D., & Norman, K. A. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/196758 · Cited by 8
(2018) · Digital Commons@Becker (Washington University School of Medicine) · doi:10.1371/journal
Research works printed but not linked: 10
- Barreto, A., Munos, R., Schaul, T., & Silver, D. (2016). Successor features for transfer in reinforcement learning . PsyArXiv. http://arxiv.org/abs/ 1606.05312
- Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). Openai gym . PsyArXiv. https://doi.org/10.48550/ arXiv.1606.01540 · arXiv
- Dayan, P., & Kakade, S. (2001). Explaining away in weight space. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in Neural Information Processing Systems (Vol. 13, pp. 451 -457). MIT Press.
- Evans, T., & Burgess, N. (2019). Explaining away in weight space. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d ' Alch é -Buc, E. Fox, & R. Garnett (Eds.), Advances in neural information processing systems 32 (pp. 451 -457). MIT Press.
- Transactions on Signal Processing , 59 (4), 1569 -1585. https://doi.org/10 .1109/TSP.2010.2102756
- Gershman, S. J. (2017a). Context-dependent learning and causal structure. Psychonomic Bulletin & Review , 24 (2), 557 -565. https://doi.org/10.3758/ s13423-016-1110-x
- Hansen, S., Dabney, W., Barreto, A., Van de Wiele, T., Warde-Farley, D., & Mnih, V. (2019). Fast task inference with variational intrinsic successor features . PsyArXiv. http://arxiv.org/abs/1906.05030 · arXiv
- Mattar, M. G., & Daw, N. D. (2017). A rational model of prioritized experience replay [Conference session]. The 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making, The University of Michigan. https://rldm.org/wp-content/uploads/2017/06/RLDM17A bstractsBooklet.pdf
- McLaren, P. L., & Mackintosh, N. J. (2000). An elemental model of associative learning: I. Latent inhibition and perceptual learning. Animal Learning & Behavior , 28 (3), 211 -246. https://doi.org/10.3758/BF03 200258 · doi:10.3758/bf03
- Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review , 87 (6), 532 -552. https://psycnet.apa.org/record/ 1981-02676-001
Cited by
Temporal context and latent state inference in the hippocampal splitter signal
Duvelle, É., Grieves, R. M., & van der Meer, M. A. A. (2022) · eLife · doi:10.7554/elife.82357 · Cited by 61
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15
Self-citationKnowledge of information cascades through social networks facilitates strategic gossip
Xia, A., Teoh, Y. Y., Nassar, M. R., Bhandari, A., & FeldmanHall, O. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02241-2 · Cited by 18
“The SR not only captures mental abstractions across other domains, such as spatial navigation 33,36 and reward learning 34 , but it has also been shown to predict people’s inferences about friendships 37,38 .”
Cognitive graphs: Representational substrates for planning.
Yoo, J., Chrastil, E. R., & Bornstein, A. M. (2024) · Decision · doi:10.1037/dec0000249 · Cited by 10
Contextual inference through flexible integration of environmental features and behavioural outcomes
Passlack, J., & MacAskill, A. F. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014093 · Cited by 2
“To infer z we implemented an existing model of the hippocampus from [24] (Figs 2B, 2D, and S1A Fig).”
Influential
Metadata
- Authors
- Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L.
- Affiliations
- University College London (GB)
- Harvard University (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Psychological Review
- Published
- 2024-03
- Volume
- 131
- Issue
- 2
- Pages
- 578-597
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- OpenAlex
- W4376131589
- Cited by
- 14
- References
- 61
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 3
- FWCI
- 1.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Psychological Review (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- Funders
- Gatsby Charitable Foundation
- Wellcome Trust (222457/Z/21/Z; 214314/Z/18/Z)
- Air Force Office of Scientific Research (FA9550-20-1-0413)
- European Research Council
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- successor representation, context-dependent learning, prediction error, uncertainty, hippocampus, contextual memory, contextual decision-making, reinforcement learning, model-based learning, task-specific representations
- MeSH terms
- Animals, Generalization, Psychological, Humans, Learning, Reinforcement, Psychology, Reward, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.