Predictive Representations: Building Blocks of Intelligence
Open accessThe DOI matches its Crossref record; 233 of the 237 references it cites were found in a registry or an index; 45 of the 78 sources its claims cite were checked for support.
Abstract
Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This review integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation and its generalizations, which have been widely applied as both engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.
Overview
CoChat Summary
TL;DR
This survey argues that predictive representations—especially the successor representation (SR) and its generalizations, successor models and successor features—cache task-relevant predictions and serve as versatile building blocks for reinforcement learning, AI, neuroscience, and cognitive science. It integrates RL theory with evidence from artificial agents, hippocampal/entorhinal function, and human learning and memory.
Key findings
- The SR caches expected discounted future state occupancy under a policy; values can be computed as an inner product between the SR and reward, giving model-free efficiency with model-based flexibility for reward changes but not transition changes. Successor models define valid probability distributions over future states, enabling continuous/high-dimensional scaling; successor features predict accumulated state features and support generalized policy improvement and the Option Keyboard for transfer.
- In AI applications, successor features and related representations support exploration, transfer, hierarchical RL, jumpy model-based RL, and multi-agent RL. For example, Hansen et al. explored Atari games without reward for 250 million time-steps and then had 100 thousand time-steps to earn reward, achieving superhuman performance across most Atari games; Thakoor et al.’s geometric policy composition enabled convergence with an order of magnitude fewer samples in four-rooms and a continuous-control maze navigation domain.
- In neuroscience, place cells are argued to correspond to columns/rows of the SR, grid cells to eigenvectors of the SR, and boundary vector cell-based successor features capture place- and grid-cell nuances. Dopamine is proposed to encode vector-valued generalized prediction errors suitable for updating successor features, and replay has been modeled using SR-based need and gain terms.
- In cognitive science, Momennejad et al.’s two-step revaluation study found humans sensitive to both reward and transition revaluation but significantly better after reward revaluation, consistent with a hybrid model-based–SR strategy; Tomov et al. found policy reuse on the first and only test trial, consistent with successor features plus generalized policy improvement; de Cothi et al. used a 10-by-10 grid with 25 maze configurations and 10 trials per configuration and found the SR agent best explained both human and rat navigation.
Methods
- Narrative survey/review, not an empirical study; it integrates reinforcement learning theory (MDPs, Bellman equations, TD learning) with artificial intelligence, neuroscience, and cognitive science.
- It formally reviews the SR, successor models, and successor features, then surveys learning algorithms and applications across exploration, transfer, hierarchical RL, multi-agent RL, hippocampal function, dopamine, revaluation, multi-task learning, associative learning, spatial navigation, and memory.
- No primary data collection or meta-analysis is reported; the paper synthesizes and interprets cited theoretical and empirical work.
Limitations
- Stated: the survey’s focus is restricted to predictive representations in reinforcement learning; it does not review predictive representations developed for language modeling, vision, and other problems, and it omits exotic variants such as the first-occupancy representation and forward-backward representation.
- Stated/evident: several reviewed approaches have limitations—e.g., the SR assumes a finite, discrete state space, is policy-dependent, and does not generalize to new environment dynamics; successor features are tied to the dynamics with which they were learned; GPI can only specify which tasks to prioritize, not their ordering; and GPC’s performance in more complex domains is unclear.
- Evident: as a survey, its conclusions depend on the reviewed studies, with no new experiments or quantitative synthesis; some cited applications and evidence come from simple, toy, or non-human domains.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
35 Valid · 7 Not valid · 32 Not checked · 99 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“If we can find such an fθ, then the resulting classifier is approximately equal to the density ratio between µπ(s̃ | s, a) and the state marginal p(s̃) (Poole et al., 2019; Zheng et al., 2023):”
§ 3 Practical learning algorithms and associated chal- > 3.2 Learning successor models
What the sources say
- Supportedcites Poole et al., 2019 source
Section 2.6 (p. 5) says a trained critic estimates the log density ratio between a conditional distribution and its marginal. Appendix D (p. 13) explains that the density ratio can be read out from the critic’s logits. The claim applies that general result to the stated state-transition conditional and state marginal; the paper does not use that specific notation. (LLM, conf=0.87)
- Disputedcites Zheng et al., 2023 sourceExperimental — accuracy not yet measured
In “A Theoretical Analysis,” the paper defines the classifier C in Eq. 12 and states in its proof of convergence that C* equals the discounted state occupancy measure pπ(s̃ | s, a), not its ratio to p(s̃). Section 3.2 instead identifies that density ratio with a normalized exponential of the optimal critic f* (Eq. 8). (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Hindsight experience replay provides a strategy for automating reward shaping.”
§ 4 Artificial intelligence applications > 4.2 Transfer
What the sources say
- Disputedcites Andrychowicz et al., 2017 sourceExperimental — accuracy not yet measured
The motivating example (p. 3) explicitly says HER is proposed as a different solution *instead of shaping the reward*. The Algorithm section (p. 4) describes replaying transitions with additional goals and recalculating rewards using an existing reward function, not automating reward shaping. (LLM, conf=0.92)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The SR, denoted Mπ, was introduced to address the transfer problem described in the pre-”
§ 2 Theory > 2.3 The successor representation
What the sources say
- Not checked — no full text of the sourcecites Dayan, 1993 source
- Partly supportedcites Gershman, 2018 sourceExperimental — accuracy not yet measured
The Introduction presents the SR as a way to balance efficiency and flexibility, and the computational-logic section says it allows values to update immediately when rewards change. That supports a connection to transferring learned information across reward changes, but our copy does not say the SR was originally introduced to address a ‘transfer problem.’ (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As we will discuss in §3.2, we can estimate the SM with density estimation techniques such as generative adversarial learning Janner et al. (2020), variational inference (Thakoor et al., 2022), and contrastive learning (Eysenbach et al., 2020; Zheng et al., 2023).”
§ 2 Theory > 2.4 Successor models: a probabilistic perspective on the SR
What the sources say
- Supportedcites Janner et al. (2020) source
The citation to Janner et al. is attached to generative adversarial learning, not to the separately cited variational and contrastive methods. In “Training without density evaluation,” the paper describes training its γ-model as a GAN; §5.1 connects that model to the normalized successor representation. (LLM, conf=0.92)
- Supportedcites Thakoor et al. (2022) source
Section 6.2 describes learning geometric horizon model densities with a conditional VAE and optimizing an evidence lower bound, supporting the variational-inference part of the sentence. Appendix A.2 relates these models to discounted future state-visitation distributions. The other techniques are attributed to different citations, not Thakoor et al. (LLM, conf=0.91)
- Not matchedcites Eysenbach et al. (2020); Zheng et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Zheng et al. (2023) show that (under some assumptions) optimizing Eq. 51 leads to the following Bellman-like update:”
§ 3 Practical learning algorithms and associated chal- > 3.2 Learning successor models
What the sources say
- Supportedcites Zheng et al., 2023 source
Appendix A (Theoretical Analysis; page unavailable) derives a classifier update from a variant of the TD InfoNCE objective and says it resembles the standard Bellman equation. Section 3.2 specifies the assumptions for that derivation. The source numbers its objective differently from the citing document’s Eq. 51, but supports the attributed result. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
105 more claims · 14 with their cited sources checked
Create a free account to see all 173 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 35
- It may state two separate claims as one. 30
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1. Collect samples with a random policy which selects between between primitive ac- > 4.4 Jumpy model-based reinforcement learning
“The agent then samples a (potentially more distal) state sn from the longer-horizon SM, µπγ , sn ∼ µπnβ (·|sn−1, an−1).”
§ 1. Collect samples with a random policy which selects between between primitive ac- > 4.4 Jumpy model-based reinforcement learning
“An SM with a shorter horizon µπβ is easier to learn but cannot sample futures as distal as µπγ”
Citations
Cites
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,459
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Place navigation impaired in rats with hippocampal lesions
Morris, R. G. M., Garrud, P., Rawlins, J. N. P., & O'Keefe, J. (1982) · Nature · doi:10.1038/297681a0 · Cited by 6,340
The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Bi, G.-Q., & Poo, M.-M. (1998) · The Journal of Neuroscience · doi:10.1523/jneurosci.18-24-10464.1998 · Cited by 4,744
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Microstructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Advances in neural information processing systems 11
Peter Sollich, A. (1999) · Cited by 3,168
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Reactivation of hippocampal ensemble memories during sleep
Wilson, M. A., & McNaughton, B. L. (1994) · Science · doi:10.1126/science.8036517 · Cited by 3,118
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Phase relationship between hippocampal place units and the EEG theta rhythm
O'Keefe, J., & Recce, M. L. (1993) · Hippocampus · doi:10.1002/hipo.450030307 · Cited by 2,552
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
The Hippocampus as a Cognitive Map
RANCK, J. B. (1979) · American Journal of Psychiatry · doi:10.1176/ajp.136.10.1353 · Cited by 2,346
Path integration and the neural basis of the 'cognitive map'
McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I., & Moser, M.-B. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1932 · Cited by 2,024
Proceedings for the 5th International Conference on Learning Representations
Srivastava, A., & Sutton, C. A. (2017) · Cited by 1,837
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Self-citationCoordinated memory replay in the visual cortex and hippocampus during sleep
Ji, D., & Wilson, M. A. (2007) · Nature Neuroscience · doi:10.1038/nn1825 · Cited by 1,669
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621
Pattern Separation in the Dentate Gyrus and CA3 of the Hippocampus
Leutgeb, J. K., Leutgeb, S., Moser, M.-B., & Moser, E. I. (2007) · Science · doi:10.1126/science.1135801 · Cited by 1,601
The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells
Muller, R., & Kubie, J. (1987) · The Journal of Neuroscience · doi:10.1523/jneurosci.07-07-01951.1987 · Cited by 1,581
Predictive coding under the free-energy principle
Friston, K., & Kiebel, S. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0300 · Cited by 1,579
Selective suppression of hippocampal ripples impairs spatial memory
Girardeau, G., Benchenane, K., Wiener, S. I., Buzsáki, G., & Zugaro, M. B. (2009) · Nature Neuroscience · doi:10.1038/nn.2384 · Cited by 1,560
Packard, M. G., & McGaugh, J. L. (1996) · Neurobiology of Learning and Memory · doi:10.1006/nlme.1996.0007 · Cited by 1,492
Conjunctive Representation of Position, Direction, and Velocity in Entorhinal Cortex
Sargolini, F., Fyhn, M., Hafting, T., McNaughton, B. L., Witter, M. P., Moser, M.-B., & Moser, E. I. (2006) · Science · doi:10.1126/science.1125572 · Cited by 1,485
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310
Using confidence bounds for exploitation-exploration trade-offs
Auer, P. L. (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944941 · Cited by 1,229
Actions and habits: The development of behavioural autonomy
Dickinson, A. (1985) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1985.0010 · Cited by 1,217
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214
Representation of Geometric Borders in the Entorhinal Cortex
Solstad, T., Boccara, C. N., Kropff, E., Moser, M.-B., & Moser, E. I. (2008) · Science · doi:10.1126/science.1166466 · Cited by 1,213
Forward and reverse hippocampal place-cell sequences during ripples
Diba, K., & Buzsáki, G. (2007) · Nature Neuroscience · doi:10.1038/nn1961 · Cited by 1,205
A Distributed Representation of Temporal Context
Howard, M. W., & Kahana, M. J. (2002) · Journal of Mathematical Psychology · doi:10.1006/jmps.2001.1388 · Cited by 1,174
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Self-citationResponses of monkey dopamine neurons during learning of behavioral reactions
Ljungberg, T., Apicella, P., & Schultz, W. (1992) · Journal of Neurophysiology · doi:10.1152/jn.1992.67.1.145 · Cited by 987
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events
Horvitz, J. C. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00019-1 · Cited by 967
Organizing conceptual knowledge in humans with a gridlike code
Constantinescu, A. O., O’Reilly, J. X., & Behrens, T. E. J. (2016) · Science · doi:10.1126/science.aaf0941 · Cited by 937
The cognitive map in humans: spatial navigation and beyond
Epstein, R. A., Patai, E. Z., Julian, J. B., & Spiers, H. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4656 · Cited by 921
Disruption of ripple‐associated hippocampal activity during rest impairs spatial learning in the rat
Ego‐Stengel, V., & Wilson, M. A. (2010) · Hippocampus · doi:10.1002/hipo.20707 · Cited by 842
Spatial selectivity of unit activity in the hippocampal granular layer
Jung, M. W., & McNaughton, B. L. (1993) · Hippocampus · doi:10.1002/hipo.450030209 · Cited by 789
Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit
Aronov, D., Nevers, R., & Tank, D. W. (2017) · Nature · doi:10.1038/nature21692 · Cited by 773
Finite scale of spatial representation in the hippocampus
Kjelstrup, K. B., Solstad, T., Brun, V. H., Hafting, T., Leutgeb, S., Witter, M. P., Moser, E. I., & Moser, M.-B. (2008) · Science · doi:10.1126/science.1157086 · Cited by 760
Boundary Vector Cells in the Subiculum of the Hippocampal Formation
Lever, C., Burton, S., Jeewajee, A., O'Keefe, J., & Burgess, N. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1319-09.2009 · Cited by 753
Stewart, N., Chater, N., & Brown, G. D. A. (2006) · Cognitive Psychology · doi:10.1016/j.cogpsych.2005.10.003 · Cited by 737
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 717
Behavioral time scale synaptic plasticity underlies CA1 place fields
Bittner, K. C., Milstein, A. D., Grienberger, C., Romani, S., & Magee, J. C. (2017) · Science · doi:10.1126/science.aan3846 · Cited by 707
The entorhinal grid map is discretized
Stensola, H., Stensola, T., Solstad, T., Frøland, K., Moser, M.-B., & Moser, E. I. (2012) · Nature · doi:10.1038/nature11649 · Cited by 701
Markus, E., Qin, Y., Leonard, B., Skaggs, W., McNaughton, B., & Barnes, C. (1995) · The Journal of Neuroscience · doi:10.1523/jneurosci.15-11-07079.1995 · Cited by 698
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693
Instrumental Responding following Reinforcer Devaluation
Adams, C. D., & Dickinson, A. (1981) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748108400816 · Cited by 687
An oscillatory interference model of grid cell firing
Burgess, N., Barry, C., & O'Keefe, J. (2007) · Hippocampus · doi:10.1002/hipo.20327 · Cited by 686
Self-citationVector-based navigation using grid-like representations in artificial agents
Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., Pritzel, A., Chadwick, M. J., Degris, T., Modayil, J., et al. (2018) · Nature · doi:10.1038/s41586-018-0102-6 · Cited by 677
Self-citationSpeed cells in the medial entorhinal cortex
Kropff, E., Carmichael, J. E., Moser, M.-B., & Moser, E. I. (2015) · Nature · doi:10.1038/nature14622 · Cited by 668
Hebbian learning and spiking neurons
Kempter, R., Gerstner, W., & van Hemmen, J. L. (1999) · Physical Review E · doi:10.1103/physreve.59.4498 · Cited by 664
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660
Self-citationVariations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 609
Predictive Representations of State
Littman, M. L., & Sutton, R. S. (2001) · neural information processing systems · Cited by 600
Neural representations of events arise from temporal community structure
Schapiro, A. C., Rogers, T. T., Cordova, N. I., Turk-Browne, N. B., & Botvinick, M. M. (2013) · Nature Neuroscience · doi:10.1038/nn.3331 · Cited by 556
Experience‐dependent modifications of hippocampal place cell firing
Bostock, E., Muller, R. U., & Kubie, J. L. (1991) · Hippocampus · doi:10.1002/hipo.450010207 · Cited by 554
Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons
Engelhard, B., Finkelstein, J., Cox, J., Fleming, W., Jang, H. J., Ornelas, S., Koay, S. A., Thiberge, S. Y., Daw, N. D., Tank, D. W., et al. (2019) · Nature · doi:10.1038/s41586-019-1261-9 · Cited by 544
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaskowski, W. (2016) · 2016 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 541
Experience-dependent rescaling of entorhinal grids
Barry, C., Hayman, R., Burgess, N., & Jeffery, K. J. (2007) · Nature Neuroscience · doi:10.1038/nn1905 · Cited by 532
Self-citationLearning and Computational Neuroscience: Foundations of Adaptive Networks
Gabriel, M., & Moore, J. (1990) · Cited by 518
Time-Derivative Models of Pavlovian Reinforcement
Sutton, R. S., & Barto, A. G. (1990) · Cited by 499
A theory of hippocampal function in memory
Rolls, E. T. (1996) · Hippocampus · doi:10.1002/(sici)1098-1063(1996)6:6<601::aid-hipo5>3.0.co;2-j · Cited by 485
Experience-Dependent Asymmetric Shape of Hippocampal Receptive Fields
Mehta, M. R., Quirk, M. C., & Wilson, M. A. (2000) · Neuron · doi:10.1016/s0896-6273(00)81072-7 · Cited by 481
On Variational Bounds of Mutual Information
Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G. (2019) · arXiv · doi:10.48550/arxiv.1905.06922 · Cited by 480
Bayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457
The StarCraft Multi-Agent Challenge
Samvelyan, M., Rashid, T., Schroeder de Witt, C., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., & Whiteson, S. (2019) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/lvzz5205 · Cited by 453
Eligibility Traces for Off-Policy Policy Evaluation
Precup, D., Sutton, R. S., & Singh, S. P. (2000) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 439
Recurrent World Models Facilitate Policy Evolution
Ha, D., & Schmidhuber, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01999 · Cited by 429
The Role of the Hippocampus in Prediction and Imagination
Buckner, R. L. (2010) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163508 · Cited by 419
The Boundary Vector Cell Model of Place Cell Firing and Spatial Memory
Barry, C., Lever, C., Hayman, R., Hartley, T., Burton, S., O'Keefe, J., Jeffery, K., & Burgess, Ν. (2006) · Reviews in the Neurosciences · doi:10.1515/revneuro.2006.17.1-2.71 · Cited by 401
Self-citationPrioritized memory access explains planning and hippocampal replay
Mattar, M. G., & Daw, N. D. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0232-z · Cited by 393
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Self-citationRelations between Pavlovian-instrumental transfer and reinforcer devaluation
Holland, P. C. (2004) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037/0097-7403.30.2.104 · Cited by 388
A cortical-hippocampal-cortical loop of information processing during memory consolidation
Rothschild, G., Eban, E., & Frank, L. M. (2017) · Nature Neuroscience · doi:10.1038/nn.4457 · Cited by 387
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386
Self-citationFoster, D. J., & Wilson, M. A. (2007) · Hippocampus · doi:10.1002/hipo.20345 · Cited by 380
A Tutorial on Thompson Sampling
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z. (2018) · now publishers, Inc. eBooks · doi:10.1561/9781680834710 · Cited by 380
A map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Ranganath, C. (2010) · Hippocampus · doi:10.1002/hipo.20852 · Cited by 375
Grid cell symmetry is shaped by environmental geometry
Krupic, J., Bauza, M., Burton, S., Barry, C., & O’Keefe, J. (2015) · Nature · doi:10.1038/nature14153 · Cited by 373
Self-citationDeep Predictive Coding Networks for Video Prediction and Unsupervised Learning
Lotter, W. E., Kreiman, G., & Cox, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08104 · Cited by 368
Contrastive Learning as Goal-Conditioned Reinforcement Learning
Eysenbach, B., Zhang, T., Levine, S., & Salakhutdinov, R. R. (2022) · neural information processing systems · doi:10.52202/068431-2580 · Cited by 351
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346
Fragmentation of grid cell maps in a multicompartment environment
Derdikman, D., Whitlock, J. R., Tsao, A., Fyhn, M., Hafting, T., Moser, M.-B., & Moser, E. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2396 · Cited by 330
Using Grid Cells for Navigation
Bush, D., Barry, C., Manson, D., & Burgess, N. (2015) · Neuron · doi:10.1016/j.neuron.2015.07.006 · Cited by 329
Self-citationPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
Self-citationHippocampal place cells construct reward related sequences through unexplored space
Ólafsdóttir, H. F., Barry, C., Saleem, A. B., Hassabis, D., & Spiers, H. J. (2015) · eLife · doi:10.7554/elife.06063 · Cited by 319
Self-citationA Multiplexed, Heterogeneous, and Adaptive Code for Navigation in Medial Entorhinal Cortex
Hardcastle, K., Maheswaranathan, N., Ganguli, S., & Giocomo, L. M. (2017) · Neuron · doi:10.1016/j.neuron.2017.03.025 · Cited by 312
Successor Features for Transfer in Reinforcement Learning
Barreto, A. D. M. S., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Silver, D., & van Hasselt, H. P. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35439.18085 · Cited by 305
Dopamine transients are sufficient and necessary for acquisition of model-based associations
Sharpe, M. J., Chang, C. Y., Liu, M. A., Batchelor, H. M., Mueller, L. E., Jones, J. L., Niv, Y., & Schoenbaum, G. (2017) · Nature Neuroscience · doi:10.1038/nn.4538 · Cited by 300
Towards a Unified Theory of State Abstraction for MDPs.
Li, L. C., Walsh, T. J., & Littman, M. L. (2006) · Cited by 293
Prediction, sequences and the hippocampus
Lisman, J., & Redish, A. D. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0316 · Cited by 287
Deep reinforcement learning with successor features for navigation across similar environments
Zhang, J., Springenberg, J. T., Boedecker, J., & Burgard, W. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206049 · Cited by 278
Space, time, and episodic memory: The hippocampus is all over the cognitive map
Ekstrom, A. D., & Ranganath, C. (2018) · Hippocampus · doi:10.1002/hipo.22750 · Cited by 251
Reminders of past choices bias decisions for reward in humans
Bornstein, A. M., Khaw, M. W., Shohamy, D., & Daw, N. D. (2017) · Nature Communications · doi:10.1038/ncomms15958 · Cited by 223
Schacter, D. L., Benoit, R. G., De Brigard, F., & Szpunar, K. K. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2013.12.008 · Cited by 211
Dopamine neurons respond to errors in the prediction of sensory features of expected rewards
Takahashi, Y. K., Batchelor, H. M., Liu, B., Khanna, A., Morales, M., & Schoenbaum, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.025 · Cited by 200
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
Self-citationTrajectory events across hippocampal place cells require previous experience
Silva, D., Feng, T., & Foster, D. J. (2015) · Nature Neuroscience · doi:10.1038/nn.4151 · Cited by 183
Bayesian approaches to associative learning: From passive to active learning
KRUSCHKE, J. K. (2008) · Learning & Behavior · doi:10.3758/lb.36.3.210 · Cited by 179
Dordek, Y., Soudry, D., Meir, R., & Derdikman, D. (2016) · eLife · doi:10.7554/elife.10094 · Cited by 175
Coordinated grid and place cell replay during rest
Ólafsdóttir, H. F., Carpenter, F., & Barry, C. (2016) · Nature Neuroscience · doi:10.1038/nn.4291 · Cited by 175
Self-citationA Unifying Probabilistic View of Associative Learning
Gershman, S. J. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004567 · Cited by 166
Self-citationLieder, F., Griffiths, T. L., & Hsu, M. (2018) · Psychological Review · doi:10.1037/rev0000074 · Cited by 166
Hippocampal remapping as hidden state inference
Sanders, H., Wilson, M. A., & Gershman, S. J. (2020) · eLife · doi:10.7554/elife.51140 · Cited by 163
Self-citationReliance on small samples, the wavy recency effect, and similarity-based learning.
Plonsky, O., Teodorescu, K., & Erev, I. (2015) · Psychological Review · doi:10.1037/a0039413 · Cited by 162
Rethinking dopamine as generalized prediction error
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2018) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 160
Self-citationLearning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy EstimationPreprint
Liu, Q., Li, L., Tang, Z., & Zhou, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12429 · Cited by 158
Ekstrom, A. D., Spiers, H. J., Bohbot, V. D., & ROSENBAUM, R. S. (2018) · Princeton University Press eBooks · doi:10.23943/9781400890460 · Cited by 155
The hippocampus supports deliberation during value-based decisions
Bakkour, A., Palombo, D. J., Zylberberg, A., Kang, Y. H., Reid, A., Verfaellie, M., Shadlen, M. N., & Shohamy, D. (2019) · eLife · doi:10.7554/elife.46080 · Cited by 153
Declarative memory is critical for sustained advantageous complex decision-making
Gupta, R., Duff, M. C., Denburg, N. L., Cohen, N. J., Bechara, A., & Tranel, D. (2009) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2009.02.007 · Cited by 145
Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions
Keiflin, R., Pribut, H. J., Shah, N. B., & Janak, P. H. (2019) · Current Biology · doi:10.1016/j.cub.2018.11.050 · Cited by 142
Spatial goal coding in the hippocampal formation
Nyberg, N., Duvelle, É., Barry, C., & Spiers, H. J. (2022) · Neuron · doi:10.1016/j.neuron.2021.12.012 · Cited by 141
Self-citationKiernan, M. J., & Westbrook, R. F. (1993) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640749308401089 · Cited by 132
Offline replay supports planning in human reinforcement learning
Momennejad, I., Otto, A. R., Daw, N. D., & Norman, K. A. (2018) · eLife · doi:10.7554/elife.32548 · Cited by 131
Visual Semantic Planning Using Deep Successor Representations
Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L., Gupta, A., Mottaghi, R., & Farhadi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.60 · Cited by 126
Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems
(1995) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.1995.525766 · Cited by 121
Winocur, G., Frankland, P. W., Sekeres, M., Fogel, S., & Moscovitch, M. (2009) · Learning & Memory · doi:10.1101/lm.1447209 · Cited by 120
Does mental context drift or shift?
DuBrow, S., Rouhani, N., Niv, Y., & Norman, K. A. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.08.003 · Cited by 118
The successor representation and temporal context
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012) · Neural Computation · doi:10.1162/neco_a_00282 · Cited by 117
Self-citationEvaluating the TD model of classical conditioning
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2012) · Learning & Behavior · doi:10.3758/s13420-012-0082-6 · Cited by 115
Neural systems supporting navigation
Spiers, H. J., & Barry, C. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.08.005 · Cited by 114
Self-citationPredictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Self-citationGutbrod, K., KROUZEL, C., Hofer, H., Müri, R. M., Perrig, W. J., & Ptak, R. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2006.01.014 · Cited by 107
Universal Successor Features ApproximatorsPreprint
Borsa, D., Barreto, A. D. M. S., Quan, J., Mankowitz, D. J., Munos, R., van Hasselt, H. P., Silver, D., & Schaul, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07626 · Cited by 106
Neural context reinstatement predicts memory misattribution
Gershman, S. J., Schapiro, A. C., Hupbach, A., & Norman, K. A. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.0096-13.2013 · Cited by 105
Self-citationPredictive Representations in Hippocampal and Prefrontal Hierarchies
Brunec, I. K., & Momennejad, I. (2022) · The Journal of Neuroscience · doi:10.1523/jneurosci.1327-21.2021 · Cited by 105
Improving Regression Performance with Distributional LossesPreprint
Imani, E., & White, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04613 · Cited by 102
Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and PlanningPreprint
Tsividis, P. A., Loula, J., Burga, J., Foss, N., Campero, A., Pouncy, T., Gershman, S. J., & Tenenbaum, J. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12544 · Cited by 101
Self-citationDecision by sampling implements efficient coding of psychoeconomic functions.
Bhui, R., & Gershman, S. J. (2018) · Psychological Review · doi:10.1037/rev0000123 · Cited by 98
Self-citationHuman Episodic Memory Retrieval Is Accompanied by a Neural Contiguity Effect
Folkerts, S., Rutishauser, U., & Howard, M. W. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.2312-17.2018 · Cited by 97
Discovery of Useful Questions as Auxiliary TasksPreprint
Veeriah, V., Hessel, M., Xu, Z., Lewis, R. L., Rajendran, J., Oh, J., van Hasselt, H. P., Silver, D., & Singh, S. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04607 · Cited by 89
Mesoaccumbal Dopamine Heterogeneity: What Do Dopamine Firing and Release Have to Do with It?
de Jong, J. W., Fraser, K. M., & Lammel, S. (2022) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-110920-011929 · Cited by 86
Optogenetic Blockade of Dopamine Transients Prevents Learning Induced by Changes in Reward Features
Chang, C. Y., Gardner, M., Di Tillio, M. G., & Schoenbaum, G. (2017) · Current Biology · doi:10.1016/j.cub.2017.09.049 · Cited by 83
Geist, M., & Pietquin, O. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3077 · Cited by 82
Abstract representations of events arise from mental errors in learning and memory
Lynn, C. W., Kahn, A. E., Nyema, N., & Bassett, D. S. (2020) · Nature Communications · doi:10.1038/s41467-020-15146-7 · Cited by 82
Deep Successor Reinforcement Learning
Kulkarni, T. D., Saeedi, A., Gautam, S., & Gershman, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02396 · Cited by 82
Self-citationRapid learning of predictive maps with STDP and theta phase precession
George, T. M., de Cothi, W., Stachenfeld, K., & Barry, C. (2023) · eLife · doi:10.7554/elife.80663 · Cited by 80
Self-citationDeforming the metric of cognitive maps distorts memory
Bellmund, J. L. S., de Cothi, W., Ruiter, T. A., Nau, M., Barry, C., & Doeller, C. F. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0767-3 · Cited by 79
Self-citationNavratilova, Z., Hoang, L. T., Schwindel, C. D., Tatsuno, M., & McNaughton, B. L. (2012) · Frontiers in Neural Circuits · doi:10.3389/fncir.2012.00006 · Cited by 78
Fast reinforcement learning with generalized policy updates
Barreto, A., Hou, S., Borsa, D., Silver, D., & Precup, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907370117 · Cited by 78
Dopamine signals as temporal difference errors: recent advances
Starkweather, C. K., & Uchida, N. (2020) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2020.08.014 · Cited by 78
Alvernhe, A., Save, E., & Poucet, B. (2011) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07653.x · Cited by 77
Dopamine neuron ensembles signal the content of sensory prediction errors
Stalnaker, T. A., Howard, J. D., Takahashi, Y. K., Gershman, S. J., Kahnt, T., & Schoenbaum, G. (2019) · eLife · doi:10.7554/elife.49315 · Cited by 74
Self-citationNeural learning rules for generating flexible predictions and computing the successor representation
Fang, C., Aronov, D., Abbott, L. F., & Mackevicius, E. L. (2022) · eLife · doi:10.7554/elife.80680 · Cited by 73
A general model of hippocampal and dorsal striatal learning and decision making
Geerts, J. P., Chersi, F., Stachenfeld, K. L., & Burgess, N. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2007981117 · Cited by 68
Memory as a Computational Resource
Dasgupta, I., & Gershman, S. J. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.12.008 · Cited by 68
Self-citationEigenoption Discovery through the Deep Successor Representation
Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G. J., & Campbell, M. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11089 · Cited by 66
Explaining Away in Weight Space
Dayan, P., & Kakade, S. M. (2000) · ScholarlyCommons (University of Pennsylvania) · Cited by 61
Hart, E. E., Sharpe, M. J., Gardner, M., & Schoenbaum, G. M. (2020) · eLife · doi:10.7554/elife.59998 · Cited by 59
Multi-task reinforcement learning in humans
Tomov, M. S., Schulz, E., & Gershman, S. J. (2021) · Nature Human Behaviour · doi:10.1038/s41562-020-01035-y · Cited by 59
Self-citationConstructing Semantic Representations From a Gradually Changing Representation of Temporal Context
Howard, M. W., Shankar, K. H., & Jagadisan, U. K. K. (2011) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2010.01112.x · Cited by 57
Competition and Cooperation Between Multiple Reinforcement Learning Systems
Kool, W., Cushman, F. A., & Gershman, S. J. (2018) · Goal-Directed Decision Making · doi:10.1016/b978-0-12-812098-9.00007-3 · Cited by 56
Self-citationPredicting the Future with Multi-scale Successor Representations
Momennejad, I., & Howard, M. W. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/449470 · Cited by 54
Interpreting hippocampal function as recoding and forecasting
Levy, W. B., Hocking, A. B., & Wu, X. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.08.005 · Cited by 53
Neurobiological successor features for spatial navigation
de Cothi, W., & Barry, C. (2020) · Hippocampus · doi:10.1002/hipo.23246 · Cited by 53
Self-citationPredictive processing in cognitive robotics: A review
Ciria, A., Schillaci, G., Pezzulo, G., Hafner, V. V., & Lara, B. (2021) · Neural Computation · doi:10.1162/neco_a_01383 · Cited by 51
Learning predictive cognitive maps with spiking neurons during behavior and replays
Bono, J., Zannone, S., Pedrosa, V., & Clopath, C. (2023) · eLife · doi:10.7554/elife.80671 · Cited by 48
Successor Features Combine Elements of Model-Free and Model-based Reinforcement LearningPreprint
Lehnert, L., & Littman, M. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11437 · Cited by 47
A unified theory for the origin of grid cells through the lens of pattern formation
Sorscher, B., Mel, G. C., Ganguli, S., & Ocko, S. A. (2019) · neural information processing systems · Cited by 47
A neurally plausible model learns successor representations in partially observable environmentsPreprint
Vértes, E., & Sahani, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09480 · Cited by 45
Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
Barreto, A. D. M. S., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D. J., Žídek, A., & Munos, R. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.22017.40802 · Cited by 45
The learning of prospective and retrospective cognitive maps within neural circuits
Namboodiri, V. M. K., & Stuber, G. D. (2021) · Neuron · doi:10.1016/j.neuron.2021.09.034 · Cited by 41
Truly Batch Apprenticeship Learning with Deep Successor Features
Lee, D., Srinivasan, S., & Doshi‐Velez, F. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/819 · Cited by 39
Actionable Neural Representations: Grid Cells from Minimal ConstraintsPreprint
Dorrell, W., Latham, P. E., Behrens, T. E. J., & Whittington, J. C. R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15563 · Cited by 38
Composing Entropic Policies using Divergence CorrectionPreprint
Hunt, J. J., Barreto, A. D. M. S., Lillicrap, T., & Heess, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02216 · Cited by 38
Tanni, S., de Cothi, W., & Barry, C. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.046 · Cited by 37
Self-citationStriatal and hippocampal contributions to flexible navigation in rats and humans
Gahnstrom, C. J., & Spiers, H. J. (2020) · Brain and Neuroscience Advances · doi:10.1177/2398212820979772 · Cited by 36
Does Zero-Shot Reinforcement Learning Exist?Preprint
Touati, A., Rapin, J., & Ollivier, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14935 · Cited by 36
A Bayesian Analysis of Dynamics in Free Recall
Socher, R., Gershman, S. J., Sederberg, P. B., Norman, K. A., Perotte, A., & Blei, D. M. (2009) · neural information processing systems · Cited by 36
Self-citationTemporal Abstraction in Reinforcement Learning with the Successor RepresentationPreprint
Machado, M. C., Barreto, A. D. M. S., Precup, D., & Bowling, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05740 · Cited by 36
Diversifying AI: Towards Creative Chess with AlphaZeroPreprint
Zahavy, T., Veeriah, V., Hou, S., Waugh, K., Lai, M., Leurent, E., Tomašev, N., Schut, L., Hassabis, D., & Singh, S. P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.09175 · Cited by 35
The Option Keyboard: Combining Skills in Reinforcement Learning
Barreto, A. D. M. S., Borsa, D., Hou, S., Comanici, G., Aygün, E., Hamel, P., Toyama, D., Hunt, J. J., Mourad, S., Silver, D., et al. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.12370.50882 · Cited by 35
Filos, A., Lyle, C., Gal, Y., Levine, S., Jaques, N., & Farquhar, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12560 · Cited by 35
APS: Active Pretraining with Successor Features
Liu, H., & Abbeel, P. (2021) · arXiv · doi:10.48550/arxiv.2108.13956 · Cited by 35
Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
Wärnberg, E., & Kumar, A. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2221994120 · Cited by 34
The context repetition effect: Predicted events are remembered better, even when they don’t happen
Smith, T. A., Hasinski, A. E., & Sederberg, P. B. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0034067 · Cited by 33
Better transfer learning with inferred successor mapsPreprint
Madarász, T. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07663 · Cited by 32
Lingering representations of stimuli influence recall organization
Chan, S. C. Y., Applegate, M. C., Morton, N. W., Polyn, S. M., & Norman, K. A. (2017) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.01.029 · Cited by 28
Successor Options: An Option Discovery Framework for Reinforcement Learning
Ramesh, R., Tomar, M., & Ravindran, B. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/458 · Cited by 27
Discovering Diverse Nearly Optimal Policies with Successor FeaturesPreprint
Zahavy, T., O’Donoghue, B., Barreto, A., Mnih, V., Flennerhag, S., & Singh, S. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00669 · Cited by 27
Generative Temporal Difference Learning for Infinite-Horizon Prediction
Janner, M., Mordatch, I., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2010.14496 · Cited by 24
Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning
Janz, D., Hron, J., Mazur, P., Hofmann, K., Hernández-Lobato, J. M., & Tschiatschek, S. (2018) · arXiv · doi:10.48550/arxiv.1810.06530 · Cited by 23
UneVEn: Universal Value Exploration for Multi-Agent Reinforcement LearningPreprint
Gupta, T., Mahajan, A., Peng, B., Böhmer, W., & Whiteson, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02974 · Cited by 22
Research works printed but not linked: 6
- Bellman, 1957, Dynamic programming
- Farebrother, 2023, Proto-value networks, Scaling representation learning with auxiliary tasks
- Fujimoto, 2021, A deep reinforcement learning approach to marginalized importance sampling with the successor representation, 3518
- Hawkins, 2004, On intelligence
- Moskovitz, 2022, A first-occupancy representation for reinforcement learning
- Wittkuhn, 2022, Statistical learning of successor representations is related to on-task replay.
Cited by
Alanazi, M., Soh, B., Samra, H. E., & Li, A. (2025) · Computers · doi:10.3390/computers14050185 · Cited by 53
“As a result, AI-driven platforms, such as ChatGPT, are increasingly being integrated into computer programming curricula to offer customised support and real-time guidance, supplementing traditional teaching methods that may struggle to provide individualised attention [3].”
Learning Temporal Relationships Between Symbols with Laplace Neural Manifolds
Howard, M. W., Esfahani, Z. G., Le, B., & Sederberg, P. B. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00230-8 · Cited by 13
Subicular spatial codes arise from predictive mappingPreprint
Bennett, L., de Cothi, W., Muessig, L., Rodrigues, F. R., Cacucci, F., Wills, T. J., Sun, Y., Giocomo, L. M., Lever, C., Poulter, S. L., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.06.622306 · Cited by 6
Self-citationNeural signatures of predictive coding underlying the acquisition of incidental sensory associationsPreprint
Greco, A., Rastelli, C., Bonetti, L., Braun, C., & Carìa, A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.16.654429 · Cited by 4
“…the neural mechanisms underlying incidental learning and for cross-modal sensory associations, showing that not only is incidental information consolidated over days, but that it continues to shape neural representations in a way consistent with predictive coding theories ( 3 , 33 , 36 , 56 , 85 ).”
Keck, J., Barry, C., Doeller, C. F., & Jost, J. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013056 · Cited by 9
Self-citation
Metadata
- Authors
- Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J.
- Affiliations
- Harvard University (US)
- Department of Cell and Developmental Biology, University College London, London WC1E 7JE, U.K. w.decothi@ucl.ac.uk
- University College London (GB)
- Corresponding author
- Carvalho, Wilka
- Tomov, Momchil S.
- Barry, Caswell
- Gershman, Samuel J.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2024-10-11
- Volume
- 36
- Issue
- 11
- Pages
- 2225-2298
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W4402046009
- Semantic Scholar
- 8814760d5b024e504d59b2849b2349bc67631536
- Cited by
- 15
- References
- 237
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 3
- FWCI
- 0.51
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Wellcome Trust (212281, 212281/Z/18/Z)
- Multidisciplinary University Research Initiative (W911NF-23-1-0277)
- Army Research Office (W911NF-23-1-0277, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- predictive representations, reinforcement learning, successor representation, adaptive behavior, prediction, cognition, intelligence
- MeSH terms
- Animals, Artificial Intelligence, Brain, Cognition, Humans, Intelligence, Models, Neurological, Reinforcement, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.