CoChat

A probabilistic successor representation for context-dependent learning

Open access

Published version of a preprint

The DOI matches its Crossref record; 61 of the 83 references it cites were found in a registry or an index, and 22 are not checked yet.

Abstract

Two of the main impediments to learning complex tasks are that relationships between different stimuli, including rewards, can be uncertain and context-dependent. Reinforcement learning (RL) provides a framework for learning, by predicting total future reward directly (model-free RL), or via predictions of future states (model-based RL). Within this framework, "successor representation" (SR) predicts total future occupancy of all states. A recent theoretical proposal suggests that the hippocampus encodes the SR in order to facilitate prediction of future reward. However, this proposal does not take into account how learning should adapt under uncertainty and switches of context. Here, we introduce a theory of learning SRs using prediction errors which includes optimally balancing uncertainty in new observations versus existing knowledge. We then generalize that approach to a multicontext setting, allowing the model to learn and maintain multiple task-specific SRs and infer which one to use at any moment based on the accuracy of its predictions. Thus, the context used for predictions can be determined by both the contents of the states themselves and the distribution of transitions between them. This probabilistic SR model captures animal behavior in tasks which require contextual memory and generalization, and unifies previous SR theory with hippocampal-dependent contextual decision-making. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1037/rev0000414

Cites

61 of 83 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 10

  1. Barreto, A., Munos, R., Schaul, T., & Silver, D. (2016). Successor features for transfer in reinforcement learning . PsyArXiv. http://arxiv.org/abs/ 1606.05312
  2. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). Openai gym . PsyArXiv. https://doi.org/10.48550/ arXiv.1606.01540 · arXiv
  3. Dayan, P., & Kakade, S. (2001). Explaining away in weight space. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in Neural Information Processing Systems (Vol. 13, pp. 451 -457). MIT Press.
  4. Evans, T., & Burgess, N. (2019). Explaining away in weight space. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d ' Alch é -Buc, E. Fox, & R. Garnett (Eds.), Advances in neural information processing systems 32 (pp. 451 -457). MIT Press.
  5. Transactions on Signal Processing , 59 (4), 1569 -1585. https://doi.org/10 .1109/TSP.2010.2102756
  6. Gershman, S. J. (2017a). Context-dependent learning and causal structure. Psychonomic Bulletin & Review , 24 (2), 557 -565. https://doi.org/10.3758/ s13423-016-1110-x
  7. Hansen, S., Dabney, W., Barreto, A., Van de Wiele, T., Warde-Farley, D., & Mnih, V. (2019). Fast task inference with variational intrinsic successor features . PsyArXiv. http://arxiv.org/abs/1906.05030 · arXiv
  8. Mattar, M. G., & Daw, N. D. (2017). A rational model of prioritized experience replay [Conference session]. The 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making, The University of Michigan. https://rldm.org/wp-content/uploads/2017/06/RLDM17A bstractsBooklet.pdf
  9. McLaren, P. L., & Mackintosh, N. J. (2000). An elemental model of associative learning: I. Latent inhibition and perceptual learning. Animal Learning & Behavior , 28 (3), 211 -246. https://doi.org/10.3758/BF03 200258 · doi:10.3758/bf03
  10. Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review , 87 (6), 532 -552. https://psycnet.apa.org/record/ 1981-02676-001

Cited by

See all 14

Metadata

Authors
Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L.
Affiliations
  • University College London (GB)
  • Harvard University (US)
  • Google DeepMind (United Kingdom) (GB)
Venue
Psychological Review
Published
2024-03
Volume
131
Issue
2
Pages
578-597
ISSN
  • 0033-295X (ISSN-L)
  • 1939-1471
OpenAlex
W4376131589
Cited by
14
References
61
Citations by year
2026: 6 · 2025: 4 · 2024: 3
FWCI
1.94
Type
journal article
Language
en
Open access
Locations
  • Psychological Review (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion, cc-by)
Funders
  • Gatsby Charitable Foundation
  • Wellcome Trust (222457/Z/21/Z; 214314/Z/18/Z)
  • Air Force Office of Scientific Research (FA9550-20-1-0413)
  • European Research Council
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
Keywords
successor representation, context-dependent learning, prediction error, uncertainty, hippocampus, contextual memory, contextual decision-making, reinforcement learning, model-based learning, task-specific representations
MeSH terms
Animals, Generalization, Psychological, Humans, Learning, Reinforcement, Psychology, Reward, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.