Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 233 of the 237 references it cites were found in a registry or an index; 45 of the 78 sources its claims cite were checked for support.

Predictive Representations: Building Blocks of Intelligence

Open access
Carvalho, Wilka, Tomov, Momchil S., de Cothi, William, Barry, Caswell, Gershman, Samuel J. · 2024 · Neural Computation

Abstract

Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This review integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation and its generalizations, which have been widely applied as both engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

Overview

CoChat Summary

TL;DR

This survey argues that predictive representations—especially the successor representation (SR) and its generalizations, successor models and successor features—cache task-relevant predictions and serve as versatile building blocks for reinforcement learning, AI, neuroscience, and cognitive science. It integrates RL theory with evidence from artificial agents, hippocampal/entorhinal function, and human learning and memory.

Key findings

  • The SR caches expected discounted future state occupancy under a policy; values can be computed as an inner product between the SR and reward, giving model-free efficiency with model-based flexibility for reward changes but not transition changes. Successor models define valid probability distributions over future states, enabling continuous/high-dimensional scaling; successor features predict accumulated state features and support generalized policy improvement and the Option Keyboard for transfer.
  • In AI applications, successor features and related representations support exploration, transfer, hierarchical RL, jumpy model-based RL, and multi-agent RL. For example, Hansen et al. explored Atari games without reward for 250 million time-steps and then had 100 thousand time-steps to earn reward, achieving superhuman performance across most Atari games; Thakoor et al.’s geometric policy composition enabled convergence with an order of magnitude fewer samples in four-rooms and a continuous-control maze navigation domain.
  • In neuroscience, place cells are argued to correspond to columns/rows of the SR, grid cells to eigenvectors of the SR, and boundary vector cell-based successor features capture place- and grid-cell nuances. Dopamine is proposed to encode vector-valued generalized prediction errors suitable for updating successor features, and replay has been modeled using SR-based need and gain terms.
  • In cognitive science, Momennejad et al.’s two-step revaluation study found humans sensitive to both reward and transition revaluation but significantly better after reward revaluation, consistent with a hybrid model-based–SR strategy; Tomov et al. found policy reuse on the first and only test trial, consistent with successor features plus generalized policy improvement; de Cothi et al. used a 10-by-10 grid with 25 maze configurations and 10 trials per configuration and found the SR agent best explained both human and rat navigation.

Methods

  • Narrative survey/review, not an empirical study; it integrates reinforcement learning theory (MDPs, Bellman equations, TD learning) with artificial intelligence, neuroscience, and cognitive science.
  • It formally reviews the SR, successor models, and successor features, then surveys learning algorithms and applications across exploration, transfer, hierarchical RL, multi-agent RL, hippocampal function, dopamine, revaluation, multi-task learning, associative learning, spatial navigation, and memory.
  • No primary data collection or meta-analysis is reported; the paper synthesizes and interprets cited theoretical and empirical work.

Limitations

  • Stated: the survey’s focus is restricted to predictive representations in reinforcement learning; it does not review predictive representations developed for language modeling, vision, and other problems, and it omits exotic variants such as the first-occupancy representation and forward-backward representation.
  • Stated/evident: several reviewed approaches have limitations—e.g., the SR assumes a finite, discrete state space, is policy-dependent, and does not generalize to new environment dynamics; successor features are tied to the dynamics with which they were learned; GPI can only specify which tasks to prioritize, not their ordering; and GPC’s performance in more complex domains is unclear.
  • Evident: as a survey, its conclusions depend on the reviewed studies, with no new experiments or quantitative synthesis; some cited applications and evidence come from simple, toy, or non-human domains.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1162/neco_a_01705

Cites

233 of 233 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

35 Valid · 7 Not valid · 32 Not checked · 99 Own results and interpretations

105 more claims · 14 with their cited sources checked

Create a free account to see all 173 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 35
  • It may state two separate claims as one. 30

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 237 references

Research works printed but not linked: 6

  1. Bellman, 1957, Dynamic programming
  2. Farebrother, 2023, Proto-value networks, Scaling representation learning with auxiliary tasks
  3. Fujimoto, 2021, A deep reinforcement learning approach to marginalized importance sampling with the successor representation, 3518
  4. Hawkins, 2004, On intelligence
  5. Moskovitz, 2022, A first-occupancy representation for reinforcement learning
  6. Wittkuhn, 2022, Statistical learning of successor representations is related to on-task replay.

Cited by

See all 15

Metadata

Authors
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J.
Affiliations
  • Harvard University (US)
  • Department of Cell and Developmental Biology, University College London, London WC1E 7JE, U.K. w.decothi@ucl.ac.uk
  • University College London (GB)
Corresponding author
  • Carvalho, Wilka
  • Tomov, Momchil S.
  • Barry, Caswell
  • Gershman, Samuel J.
Venue
Neural Computation
Publisher
MIT Press
Published
2024-10-11
Volume
36
Issue
11
Pages
2225-2298
ISSN
  • 0899-7667 (ISSN-L)
  • 1530-888X
OpenAlex
W4402046009
Cited by
15
References
237
Citations by year
2026: 2 · 2025: 10 · 2024: 3
FWCI
0.51
Type
journal article
Language
en
Open access
Locations
  • Neural Computation (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Wellcome Trust (212281, 212281/Z/18/Z)
  • Multidisciplinary University Research Initiative (W911NF-23-1-0277)
  • Army Research Office (W911NF-23-1-0277, W911NF)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
Keywords
predictive representations, reinforcement learning, successor representation, adaptive behavior, prediction, cognition, intelligence
MeSH terms
Animals, Artificial Intelligence, Brain, Cognition, Humans, Intelligence, Models, Neurological, Reinforcement, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Predictive Representations: Building Blocks of Intelligence. https://app.cochat.ai/papers/10.1162/neco_a_01705
The paper
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024). Predictive Representations: Building Blocks of Intelligence. Neural Computation. https://doi.org/10.1162/neco_a_01705

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.