The DOI matches its Crossref record; 233 of the 237 references it cites were found in a registry or an index; 45 of the 78 sources its claims cite were checked for support.
Predictive Representations: Building Blocks of Intelligence
Open accessAbstract
Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This review integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation and its generalizations, which have been widely applied as both engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.
Overview
CoChat Summary
TL;DR
This survey argues that predictive representations—especially the successor representation (SR) and its generalizations, successor models and successor features—cache task-relevant predictions and serve as versatile building blocks for reinforcement learning, AI, neuroscience, and cognitive science. It integrates RL theory with evidence from artificial agents, hippocampal/entorhinal function, and human learning and memory.
Key findings
- The SR caches expected discounted future state occupancy under a policy; values can be computed as an inner product between the SR and reward, giving model-free efficiency with model-based flexibility for reward changes but not transition changes. Successor models define valid probability distributions over future states, enabling continuous/high-dimensional scaling; successor features predict accumulated state features and support generalized policy improvement and the Option Keyboard for transfer.
- In AI applications, successor features and related representations support exploration, transfer, hierarchical RL, jumpy model-based RL, and multi-agent RL. For example, Hansen et al. explored Atari games without reward for 250 million time-steps and then had 100 thousand time-steps to earn reward, achieving superhuman performance across most Atari games; Thakoor et al.’s geometric policy composition enabled convergence with an order of magnitude fewer samples in four-rooms and a continuous-control maze navigation domain.
- In neuroscience, place cells are argued to correspond to columns/rows of the SR, grid cells to eigenvectors of the SR, and boundary vector cell-based successor features capture place- and grid-cell nuances. Dopamine is proposed to encode vector-valued generalized prediction errors suitable for updating successor features, and replay has been modeled using SR-based need and gain terms.
- In cognitive science, Momennejad et al.’s two-step revaluation study found humans sensitive to both reward and transition revaluation but significantly better after reward revaluation, consistent with a hybrid model-based–SR strategy; Tomov et al. found policy reuse on the first and only test trial, consistent with successor features plus generalized policy improvement; de Cothi et al. used a 10-by-10 grid with 25 maze configurations and 10 trials per configuration and found the SR agent best explained both human and rat navigation.
Methods
- Narrative survey/review, not an empirical study; it integrates reinforcement learning theory (MDPs, Bellman equations, TD learning) with artificial intelligence, neuroscience, and cognitive science.
- It formally reviews the SR, successor models, and successor features, then surveys learning algorithms and applications across exploration, transfer, hierarchical RL, multi-agent RL, hippocampal function, dopamine, revaluation, multi-task learning, associative learning, spatial navigation, and memory.
- No primary data collection or meta-analysis is reported; the paper synthesizes and interprets cited theoretical and empirical work.
Limitations
- Stated: the survey’s focus is restricted to predictive representations in reinforcement learning; it does not review predictive representations developed for language modeling, vision, and other problems, and it omits exotic variants such as the first-occupancy representation and forward-backward representation.
- Stated/evident: several reviewed approaches have limitations—e.g., the SR assumes a finite, discrete state space, is policy-dependent, and does not generalize to new environment dynamics; successor features are tied to the dynamics with which they were learned; GPI can only specify which tasks to prioritize, not their ordering; and GPC’s performance in more complex domains is unclear.
- Evident: as a survey, its conclusions depend on the reviewed studies, with no new experiments or quantitative synthesis; some cited applications and evidence come from simple, toy, or non-human domains.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
35 Valid · 7 Not valid · 32 Not checked · 99 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“If we can find such an fθ, then the resulting classifier is approximately equal to the density ratio between µπ(s̃ | s, a) and the state marginal p(s̃) (Poole et al., 2019; Zheng et al., 2023):”
§ 3 Practical learning algorithms and associated chal- > 3.2 Learning successor models
What the sources say
- Supportedcites Poole et al., 2019 source
Section 2.6 (p. 5) says a trained critic estimates the log density ratio between a conditional distribution and its marginal. Appendix D (p. 13) explains that the density ratio can be read out from the critic’s logits. The claim applies that general result to the stated state-transition conditional and state marginal; the paper does not use that specific notation. (LLM, conf=0.87)
- Disputedcites Zheng et al., 2023 sourceExperimental — accuracy not yet measured
In “A Theoretical Analysis,” the paper defines the classifier C in Eq. 12 and states in its proof of convergence that C* equals the discounted state occupancy measure pπ(s̃ | s, a), not its ratio to p(s̃). Section 3.2 instead identifies that density ratio with a normalized exponential of the optimal critic f* (Eq. 8). (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Hindsight experience replay provides a strategy for automating reward shaping.”
§ 4 Artificial intelligence applications > 4.2 Transfer
What the sources say
- Disputedcites Andrychowicz et al., 2017 sourceExperimental — accuracy not yet measured
The motivating example (p. 3) explicitly says HER is proposed as a different solution *instead of shaping the reward*. The Algorithm section (p. 4) describes replaying transitions with additional goals and recalculating rewards using an existing reward function, not automating reward shaping. (LLM, conf=0.92)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The SR, denoted Mπ, was introduced to address the transfer problem described in the pre-”
§ 2 Theory > 2.3 The successor representation
What the sources say
- Not checked — no full text of the sourcecites Dayan, 1993 source
- Partly supportedcites Gershman, 2018 sourceExperimental — accuracy not yet measured
The Introduction presents the SR as a way to balance efficiency and flexibility, and the computational-logic section says it allows values to update immediately when rewards change. That supports a connection to transferring learned information across reward changes, but our copy does not say the SR was originally introduced to address a ‘transfer problem.’ (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As we will discuss in §3.2, we can estimate the SM with density estimation techniques such as generative adversarial learning Janner et al. (2020), variational inference (Thakoor et al., 2022), and contrastive learning (Eysenbach et al., 2020; Zheng et al., 2023).”
§ 2 Theory > 2.4 Successor models: a probabilistic perspective on the SR
What the sources say
- Supportedcites Janner et al. (2020) source
The citation to Janner et al. is attached to generative adversarial learning, not to the separately cited variational and contrastive methods. In “Training without density evaluation,” the paper describes training its γ-model as a GAN; §5.1 connects that model to the normalized successor representation. (LLM, conf=0.92)
- Supportedcites Thakoor et al. (2022) source
Section 6.2 describes learning geometric horizon model densities with a conditional VAE and optimizing an evidence lower bound, supporting the variational-inference part of the sentence. Appendix A.2 relates these models to discounted future state-visitation distributions. The other techniques are attributed to different citations, not Thakoor et al. (LLM, conf=0.91)
- Not matchedcites Eysenbach et al. (2020); Zheng et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Zheng et al. (2023) show that (under some assumptions) optimizing Eq. 51 leads to the following Bellman-like update:”
§ 3 Practical learning algorithms and associated chal- > 3.2 Learning successor models
What the sources say
- Supportedcites Zheng et al., 2023 source
Appendix A (Theoretical Analysis; page unavailable) derives a classifier update from a variant of the TD InfoNCE objective and says it resembles the standard Bellman equation. Section 3.2 specifies the assumptions for that derivation. The source numbers its objective differently from the citing document’s Eq. 51, but supports the attributed result. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
105 more claims · 14 with their cited sources checked
Create a free account to see all 173 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 30
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1. Collect samples with a random policy which selects between between primitive ac- > 4.4 Jumpy model-based reinforcement learning
“The agent then samples a (potentially more distal) state sn from the longer-horizon SM, µπγ , sn ∼ µπnβ (·|sn−1, an−1).”
§ 1. Collect samples with a random policy which selects between between primitive ac- > 4.4 Jumpy model-based reinforcement learning
“An SM with a shorter horizon µπβ is easier to learn but cannot sample futures as distal as µπγ”
Citations
Cites
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,462
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Research works printed but not linked: 6
- Bellman, 1957, Dynamic programming
- Farebrother, 2023, Proto-value networks, Scaling representation learning with auxiliary tasks
- Fujimoto, 2021, A deep reinforcement learning approach to marginalized importance sampling with the successor representation, 3518
- Hawkins, 2004, On intelligence
- Moskovitz, 2022, A first-occupancy representation for reinforcement learning
- Wittkuhn, 2022, Statistical learning of successor representations is related to on-task replay.
Cited by
Alanazi, M., Soh, B., Samra, H. E., & Li, A. (2025) · Computers · doi:10.3390/computers14050185 · Cited by 53
“As a result, AI-driven platforms, such as ChatGPT, are increasingly being integrated into computer programming curricula to offer customised support and real-time guidance, supplementing traditional teaching methods that may struggle to provide individualised attention [3].”
Learning Temporal Relationships Between Symbols with Laplace Neural Manifolds
Howard, M. W., Esfahani, Z. G., Le, B., & Sederberg, P. B. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00230-8 · Cited by 13
Subicular spatial codes arise from predictive mappingPreprint
Bennett, L., de Cothi, W., Muessig, L., Rodrigues, F. R., Cacucci, F., Wills, T. J., Sun, Y., Giocomo, L. M., Lever, C., Poulter, S. L., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.06.622306 · Cited by 6
Self-citationNeural signatures of predictive coding underlying the acquisition of incidental sensory associationsPreprint
Greco, A., Rastelli, C., Bonetti, L., Braun, C., & Carìa, A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.16.654429 · Cited by 4
“…the neural mechanisms underlying incidental learning and for cross-modal sensory associations, showing that not only is incidental information consolidated over days, but that it continues to shape neural representations in a way consistent with predictive coding theories ( 3 , 33 , 36 , 56 , 85 ).”
Keck, J., Barry, C., Doeller, C. F., & Jost, J. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013056 · Cited by 9
Self-citation
Metadata
- Authors
- Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J.
- Affiliations
- Harvard University (US)
- Department of Cell and Developmental Biology, University College London, London WC1E 7JE, U.K. w.decothi@ucl.ac.uk
- University College London (GB)
- Corresponding author
- Carvalho, Wilka
- Tomov, Momchil S.
- Barry, Caswell
- Gershman, Samuel J.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2024-10-11
- Volume
- 36
- Issue
- 11
- Pages
- 2225-2298
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W4402046009
- Semantic Scholar
- 8814760d5b024e504d59b2849b2349bc67631536
- Cited by
- 15
- References
- 237
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 3
- FWCI
- 0.51
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Wellcome Trust (212281, 212281/Z/18/Z)
- Multidisciplinary University Research Initiative (W911NF-23-1-0277)
- Army Research Office (W911NF-23-1-0277, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- predictive representations, reinforcement learning, successor representation, adaptive behavior, prediction, cognition, intelligence
- MeSH terms
- Animals, Artificial Intelligence, Brain, Cognition, Humans, Intelligence, Models, Neurological, Reinforcement, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Predictive Representations: Building Blocks of Intelligence. https://app.cochat.ai/papers/10.1162/neco_a_01705
- The paper
- Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024). Predictive Representations: Building Blocks of Intelligence. Neural Computation. https://doi.org/10.1162/neco_a_01705
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.