Predicting the Future with Multi-scale Successor Representations
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The successor representation (SR) is a candidate principle for generalization in reinforcement learning, computational accounts of memory, and the structure of neural representations in the hippocampus. Given a sequence of states, the SR learns a predictive representation for every given state that encodes how often, on average, each upcoming state is expected to be visited, even if it is multiple steps ahead. A discount or scale parameter determines how many steps into the future SR’s generalizations reach, enabling rapid value computation, subgoal discovery, and flexible decision-making in large trees. However, SR with a single scale could discard information for predicting both the sequential order of and the distance between states, which are common problems in navigation for animals and artificial agents. Here we propose a solution: an ensemble of SRs with multiple scales. We show that the derivative of multi-scale SR can reconstruct both the sequence of expected future states and estimate distance to goal. This derivative can be computed linearly: we show that a multi-scale SR ensemble is the Laplace transform of future states, and the inverse of this Laplace transform is a biologically plausible linear estimation of the derivative. Multi-scale SR and its derivative could lead to a common principle for how the medial temporal lobe supports both map-based and vector-based navigation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Marr, D., & Hildreth, E. (1980) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1980.0020 · Cited by 6,510
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 1×
“In the visual system it has been understood that general computational principles could enable learning of center-surround receptive fields from natural statistics (Bell & Sejnowski, 1997; Olshausen & Field, 1996).”
BackgroundThe “independent components” of natural scenes are edge filters
Bell, A. J., & Sejnowski, T. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00121-1 · Cited by 2,201
Internally Generated Cell Assembly Sequences in the Rat Hippocampus
Pastalkova, E., Itskov, V., Amarasingham, A., & Buzsáki, G. (2008) · Science · doi:10.1126/science.1159775 · Cited by 1,301
Hippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216
Hippocampal ‘time cells’ bridge the gap in memory for discontiguous events
MacDonald, C. J., Lepage, K. Q., Eden, U. T., & Eichenbaum, H. (2011) · Neuron · doi:10.1016/j.neuron.2011.07.012 · Cited by 1,193 · cited 2×
“…referred to as time cells have been observed in many brain regions (Pastalkova, Itskov, Amarasingham, & Buzsaki, 2008; Jin, Fujii, & Graybiel, 2009; MacDonald, Lepage, Eden, & Eichenbaum, 2011; Kraus et al., 2015; Mello, Soares, & Paton, 2015; Akhlaghpour et al., 2016; Tiganj, Kim, Jung, & Howard,…”
BackgroundNeural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003 · cited 1×
“( Johnson & Redish, 2007; Pfeiffer & Foster, 2013).”
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit
Aronov, D., Nevers, R., & Tank, D. W. (2017) · Nature · doi:10.1038/nature21692 · Cited by 773
Neural representations of events arise from temporal community structure
Schapiro, A. C., Rogers, T. T., Cordova, N. I., Turk-Browne, N. B., & Botvinick, M. M. (2013) · Nature Neuroscience · doi:10.1038/nn.3331 · Cited by 556
Integrating time from experience in the lateral entorhinal cortex
Tsao, A., Sugar, J., Lu, L., Wang, C., Knierim, J. J., Moser, M.-B., & Moser, E. I. (2018) · Nature · doi:10.1038/s41586-018-0459-6 · Cited by 490 · cited 1×
“On another note, a recent study has shown that temporal information is robustly encoded across time scales (seconds to hours) in lateral entorhinal cortex populations (Tsao et al., 2018), while similar information was not found in either CA3-CA1 nor in the medial entorhinal cortex.”
BackgroundA Dedicated Population for Reward Coding in the Hippocampus
Gauthier, J. L., & Tank, D. W. (2018) · Neuron · doi:10.1016/j.neuron.2018.06.008 · Cited by 394 · cited 6×
“Such an estimate would resemble distance to goal cells, which fire at particular distances to a goal location (Eichenbaum et al., 1987; Sarel, Finkelstein, Las, & Ulanovsky, 2017c; Gauthier & Tank, 2018; Qasim et al., 2018).”
BackgroundResultInfluentialA map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Social place-cells in the bat hippocampus
Omer, D. B., Maimon, S. R., Las, L., & Ulanovsky, N. (2018) · Science · doi:10.1126/science.aao3474 · Cited by 280 · cited 1×
“It has also been shown that the so called place cells also respond to non-spatial relations, e.g., sonic and social state-spaces (Aronov et al., 2017; Omer, Maimon, Las, & Ulanovsky, 2018).”
BackgroundNeural representation of time in cortico-basal ganglia circuits
Jin, D. Z., Fujii, N., & Graybiel, A. M. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0909881106 · Cited by 246
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · cited 3×
“In contrast, SR offers a more computationally efficient solution to the RL problem by storing generalizations (temporal abstractions) of multi-step relationships between states (Dayan, 1993; Gershman, 2018).”
BackgroundMethodStimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148
The successor representation and temporal context
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012) · Neural Computation · doi:10.1162/neco_a_00282 · Cited by 117 · cited 2×
“, 2017), computational accounts of episodic memory and temporal context (Gershman et al., 2012), and predictive representations in place cells and grid cells in the hippocampus and medial entorhinal cortex (Stachenfeld et al.”
BackgroundThe context repetition effect: Predicted events are remembered better, even when they don’t happen
Smith, T. A., Hasinski, A. E., & Sederberg, P. B. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0034067 · Cited by 33
Cited by
Learning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Self-citationSpatial goal coding in the hippocampal formation
Nyberg, N., Duvelle, É., Barry, C., & Spiers, H. J. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.012 · Cited by 141
George, D., Rikhye, R. V., Gothoskar, N., Guntupalli, J. S., Dedieu, A., & Lázaro-Gredilla, M. (2021) · Nature Communications · doi:10.1038/s41467-021-22559-5 · Cited by 109
Predictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15
Metadata
- Authors
- Momennejad, I., & Howard, M. W.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-10-22
- OpenAlex
- W2897698970
- Cited by
- 54
- References
- 20
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.