The DOI matches its Crossref record; 61 of the 83 references it cites were found in a registry or an index, and 22 are not checked yet.
A probabilistic successor representation for context-dependent learning
Open accessPublished version of a preprint
Abstract
Two of the main impediments to learning complex tasks are that relationships between different stimuli, including rewards, can be uncertain and context-dependent. Reinforcement learning (RL) provides a framework for learning, by predicting total future reward directly (model-free RL), or via predictions of future states (model-based RL). Within this framework, "successor representation" (SR) predicts total future occupancy of all states. A recent theoretical proposal suggests that the hippocampus encodes the SR in order to facilitate prediction of future reward. However, this proposal does not take into account how learning should adapt under uncertainty and switches of context. Here, we introduce a theory of learning SRs using prediction errors which includes optimally balancing uncertainty in new observations versus existing knowledge. We then generalize that approach to a multicontext setting, allowing the model to learn and maintain multiple task-specific SRs and infer which one to use at any moment based on the accuracy of its predictions. Thus, the context used for predictions can be determined by both the contents of the states themselves and the distribution of transitions between them. This probabilistic SR model captures animal behavior in tasks which require contextual memory and generalization, and unifies previous SR theory with hippocampal-dependent contextual decision-making. (PsycInfo Database Record (c) 2024 APA, all rights reserved).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“rather than encoding a single monolithic predictive map of the environment, the hippocampus encodes multiple separate predictive maps, each associated with its own context.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Both the context and the predictive map are inferred using Bayesian inference”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“A new context is inferred when the current predictive map fails to account for current observations.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“learning of the predictive map is modulated by uncertainty, and supports non-local updating.”
From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Geerts et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A New Approach to Linear Filtering and Prediction Problems
Kalman, R. E. (1960) · Journal of Basic Engineering · doi:10.1115/1.3662552 · Cited by 31,751
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 2×
“TD learning of the value function (using reward rather than state prediction errors) is a popular model of learning in the striatum (Schultz et al., 1997), and TD algorithms can be implemented in biologically realistic spiking networks (Bono et al.”
BackgroundThe hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 1×
“…a rich body of work in psychology and neuroscience suggesting that multiple learning systems operate in parallel, competing for control of behavior (Daw et al., 2005; Geerts et al., 2020) and that there are alternative approaches that combine aspects of model-based and model-free learning…”
BackgroundThe hippocampal-VTA loop: controlling the entry of information into long-term memory
Lisman, J. E., & Grace, A. A. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.002 · Cited by 1,983
Context and Behavioral Processes in Extinction: Table 1.
Bouton, M. E. (2004) · Learning & Memory · doi:10.1101/lm.78804 · Cited by 1,808 · cited 1×
“There is plentiful evidence that animals can deal flexibly with change, and understanding how context modulates behavior is a longstanding topic of interest in psychology and neuroscience (Bouton, 2004; Bouton & Bolles, 1979; Gershman, 2017a; Gershman et al., 2010; Heald et al., 2021).”
BackgroundKhalifa, A. A. Z., Ichikawa, M., Dai, D., Kubo, S., Black, C. S., Peri, K., McAlear, T. S., Veyron, S., Yang, S. K., Vargas, J., et al. (2020) · eLife · doi:10.7554/elife.52760 · Cited by 1,676
Bayesian Reasoning and Machine Learning
Barber, D. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804779 · Cited by 1,631
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 1×
“This is different from experience replay, in which experienced sequences of states are replayed (Lin, 1992), and simulated experience, in which possible experienced sequences are generated from a transition model (Momennejad, Otto, et al., 2017; Sutton, 1991); rather, multiple states are…”
Background
Research works printed but not linked: 10
- Barreto, A., Munos, R., Schaul, T., & Silver, D. (2016). Successor features for transfer in reinforcement learning . PsyArXiv. http://arxiv.org/abs/ 1606.05312
- Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). Openai gym . PsyArXiv. https://doi.org/10.48550/ arXiv.1606.01540 · arXiv
- Dayan, P., & Kakade, S. (2001). Explaining away in weight space. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in Neural Information Processing Systems (Vol. 13, pp. 451 -457). MIT Press.
- Evans, T., & Burgess, N. (2019). Explaining away in weight space. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d ' Alch é -Buc, E. Fox, & R. Garnett (Eds.), Advances in neural information processing systems 32 (pp. 451 -457). MIT Press.
- Transactions on Signal Processing , 59 (4), 1569 -1585. https://doi.org/10 .1109/TSP.2010.2102756
- Gershman, S. J. (2017a). Context-dependent learning and causal structure. Psychonomic Bulletin & Review , 24 (2), 557 -565. https://doi.org/10.3758/ s13423-016-1110-x
- Hansen, S., Dabney, W., Barreto, A., Van de Wiele, T., Warde-Farley, D., & Mnih, V. (2019). Fast task inference with variational intrinsic successor features . PsyArXiv. http://arxiv.org/abs/1906.05030 · arXiv
- Mattar, M. G., & Daw, N. D. (2017). A rational model of prioritized experience replay [Conference session]. The 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making, The University of Michigan. https://rldm.org/wp-content/uploads/2017/06/RLDM17A bstractsBooklet.pdf
- McLaren, P. L., & Mackintosh, N. J. (2000). An elemental model of associative learning: I. Latent inhibition and perceptual learning. Animal Learning & Behavior , 28 (3), 211 -246. https://doi.org/10.3758/BF03 200258 · doi:10.3758/bf03
- Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review , 87 (6), 532 -552. https://psycnet.apa.org/record/ 1981-02676-001
Cited by
Temporal context and latent state inference in the hippocampal splitter signal
Duvelle, É., Grieves, R. M., & van der Meer, M. A. A. (2022) · eLife · doi:10.7554/elife.82357 · Cited by 61
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15
Self-citationKnowledge of information cascades through social networks facilitates strategic gossip
Xia, A., Teoh, Y. Y., Nassar, M. R., Bhandari, A., & FeldmanHall, O. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02241-2 · Cited by 18
“The SR not only captures mental abstractions across other domains, such as spatial navigation 33,36 and reward learning 34 , but it has also been shown to predict people’s inferences about friendships 37,38 .”
Cognitive graphs: Representational substrates for planning.
Yoo, J., Chrastil, E. R., & Bornstein, A. M. (2024) · Decision · doi:10.1037/dec0000249 · Cited by 10
Contextual inference through flexible integration of environmental features and behavioural outcomes
Passlack, J., & MacAskill, A. F. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014093 · Cited by 2
“To infer z we implemented an existing model of the hippocampus from [24] (Figs 2B, 2D, and S1A Fig).”
Influential
Metadata
- Authors
- Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L.
- Affiliations
- University College London (GB)
- Harvard University (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Psychological Review
- Published
- 2024-03
- Volume
- 131
- Issue
- 2
- Pages
- 578-597
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- OpenAlex
- W4376131589
- Cited by
- 14
- References
- 61
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 3
- FWCI
- 1.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Psychological Review (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- Funders
- Gatsby Charitable Foundation
- Wellcome Trust (222457/Z/21/Z; 214314/Z/18/Z)
- Air Force Office of Scientific Research (FA9550-20-1-0413)
- European Research Council
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- successor representation, context-dependent learning, prediction error, uncertainty, hippocampus, contextual memory, contextual decision-making, reinforcement learning, model-based learning, task-specific representations
- MeSH terms
- Animals, Generalization, Psychological, Humans, Learning, Reinforcement, Psychology, Reward, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A probabilistic successor representation for context-dependent learning. https://app.cochat.ai/papers/10.1037/rev0000414
- The paper
- Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L. (2024). A probabilistic successor representation for context-dependent learning. Psychological Review. https://doi.org/10.1037/rev0000414
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.