Predicting the Future with Multi-scale Successor Representations
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The successor representation (SR) is a candidate principle for generalization in reinforcement learning, computational accounts of memory, and the structure of neural representations in the hippocampus. Given a sequence of states, the SR learns a predictive representation for every given state that encodes how often, on average, each upcoming state is expected to be visited, even if it is multiple steps ahead. A discount or scale parameter determines how many steps into the future SR’s generalizations reach, enabling rapid value computation, subgoal discovery, and flexible decision-making in large trees. However, SR with a single scale could discard information for predicting both the sequential order of and the distance between states, which are common problems in navigation for animals and artificial agents. Here we propose a solution: an ensemble of SRs with multiple scales. We show that the derivative of multi-scale SR can reconstruct both the sequence of expected future states and estimate distance to goal. This derivative can be computed linearly: we show that a multi-scale SR ensemble is the Laplace transform of future states, and the inverse of this Laplace transform is a biologically plausible linear estimation of the derivative. Multi-scale SR and its derivative could lead to a common principle for how the medial temporal lobe supports both map-based and vector-based navigation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Marr, D., & Hildreth, E. (1980) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1980.0020 · Cited by 6,510
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 1×
“In the visual system it has been understood that general computational principles could enable learning of center-surround receptive fields from natural statistics (Bell & Sejnowski, 1997; Olshausen & Field, 1996).”
BackgroundThe “independent components” of natural scenes are edge filters
Bell, A. J., & Sejnowski, T. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00121-1 · Cited by 2,201
Internally Generated Cell Assembly Sequences in the Rat Hippocampus
Pastalkova, E., Itskov, V., Amarasingham, A., & Buzsáki, G. (2008) · Science · doi:10.1126/science.1159775 · Cited by 1,301
Hippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216
Hippocampal ‘time cells’ bridge the gap in memory for discontiguous events
MacDonald, C. J., Lepage, K. Q., Eden, U. T., & Eichenbaum, H. (2011) · Neuron · doi:10.1016/j.neuron.2011.07.012 · Cited by 1,193 · cited 2×
“…referred to as time cells have been observed in many brain regions (Pastalkova, Itskov, Amarasingham, & Buzsaki, 2008; Jin, Fujii, & Graybiel, 2009; MacDonald, Lepage, Eden, & Eichenbaum, 2011; Kraus et al., 2015; Mello, Soares, & Paton, 2015; Akhlaghpour et al., 2016; Tiganj, Kim, Jung, & Howard,…”
BackgroundNeural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · cited 1×
“( Johnson & Redish, 2007; Pfeiffer & Foster, 2013).”
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit
Aronov, D., Nevers, R., & Tank, D. W. (2017) · Nature · doi:10.1038/nature21692 · Cited by 773
Cited by
Learning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Self-citationSpatial goal coding in the hippocampal formation
Nyberg, N., Duvelle, É., Barry, C., & Spiers, H. J. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.012 · Cited by 141
George, D., Rikhye, R. V., Gothoskar, N., Guntupalli, J. S., Dedieu, A., & Lázaro-Gredilla, M. (2021) · Nature Communications · doi:10.1038/s41467-021-22559-5 · Cited by 109
Predictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15
Metadata
- Authors
- Momennejad, I., & Howard, M. W.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-10-22
- OpenAlex
- W2897698970
- Cited by
- 54
- References
- 20
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.