Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) provides an influential characterization of the brain's mechanisms for learning to make advantageous choices. An important problem, though, is how complex tasks can be represented in a way that enables efficient learning. We consider this problem through the lens of spatial navigation, examining how two of the brain's location representations--hippocampal place cells and entorhinal grid cells--are adapted to serve as basis functions for approximating value over space for RL. Although much previous work has focused on these systems' roles in combining upstream sensory cues to track location, revisiting these representations with a focus on how they support this downstream decision function offers complementary insights into their characteristics. Rather than localization, the key problem in learning is generalization between past and present situations, which may not match perfectly. Accordingly, although neural populations collectively offer a precise representation of position, our simulations of navigational tasks verify the suggestion that RL gains efficiency from the more diffuse tuning of individual neurons, which allows learning about rewards to generalize over longer distances given fewer training experiences. However, work on generalization in RL suggests the underlying representation should respect the environment's layout. In particular, although it is often assumed that neurons track location in Euclidean coordinates (that a place cell's activity declines "as the crow flies" away from its peak), the relevant metric for value is geodesic: the distance along a path, around any obstacles. We formalize this intuition and present simulations showing how Euclidean, but not geodesic, representations can interfere with RL by generalizing inappropriately across barriers. Our proposal that place and grid responses should be modulated by geodesic distances suggests novel predictions about how obstacles should affect spatial firing fields, which provides a new viewpoint on data concerning both spatial codes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“We use a simple temporal-difference algorithm with eligibility traces [8,85] to learn weights.”
MethodMicrostructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationPath integration and the neural basis of the 'cognitive map'
McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I., & Moser, M.-B. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1932 · Cited by 2,024
Place Cells, Grid Cells, and the Brain's Spatial Representation System
Moser, E. I., Kropff, E., & Moser, M.-B. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.061307.090723 · Cited by 1,934 · cited 1×
“However, in the present work our goal is to investigate the brain’s spatial representations through the lens of their downstream computations; thus, in contrast to much work on the hippocampal system [12,35,36,40,43,44,47,48,62,63] we do not focus on the ‘‘upstream’’ computations by which the grid or place representat…”
BackgroundReverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727 · cited 1×
“Second, across algorithms, TD-like learning mechanisms also likely interact with additional ones in the brain, and the core considerations we elucidate about efficient generalization due to appropriate state space representations crosscut these distinctions. For instance, value functions may also be updated using repl…”
BackgroundThe effects of changes in the environment on the spatial firing of hippocampal complex-spike cells
Muller, R., & Kubie, J. (1987) · The Journal of Neuroscience · doi:10.1523/jneurosci.07-07-01951.1987 · Cited by 1,581
Conjunctive Representation of Position, Direction, and Velocity in Entorhinal Cortex
Sargolini, F., Fyhn, M., Hafting, T., McNaughton, B. L., Witter, M. P., Moser, M.-B., & Moser, E. I. (2006) · Science · doi:10.1126/science.1125572 · Cited by 1,485
Spatial representation in the entorhinal cortex
Fyhn, M., Molden, S., Witter, M. P., Moser, E. I., & Moser, M.-B. (2004) · Science · doi:10.1126/science.1099901 · Cited by 1,339 · cited 1×
“Grid cells are most commonly reported in the superficial layers (II–III) of dMEC, which project directly to hippocampus [52] though they have also been reported in deep layers [53,54], where intracortical Author Summary”
BackgroundModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Geometric determinants of the place fields of hippocampal neurons
O'Keefe, J., & Burgess, N. (1996) · Nature · doi:10.1038/381425a0 · Cited by 1,166
McNaughton, B. L., Barnes, C. A., & O'Keefe, J. (1983) · Experimental Brain Research · doi:10.1007/bf00237147 · Cited by 1,071
Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point
Johnson, A., & Redish, A. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.3761-07.2007 · Cited by 1,003
Path integration and cognitive mapping in a continuous attractor neural network model
Samsonovich, A., & McNaughton, B. L. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-15-05900.1997 · Cited by 989
Doya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642
Development of the Hippocampal Cognitive Map in Preweanling Rats
Wills, T. J., Cacucci, F., Burgess, N., & O'Keefe, J. (2010) · Science · doi:10.1126/science.1188224 · Cited by 616 · cited 1×
“Note that our model’s geometric predictions about how the grid cell representation should behave do not depend on the idea that it serves as a direct substrate for value learning: since the grid cell representation is thought to serve as a precursor of the place cell representation (though see [59,60]), it would be li…”
BackgroundDecision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Self-citationFrom grid cells to place cells: A mathematical model
Solstad, T., Moser, E. I., & Einevoll, G. T. (2006) · Hippocampus · doi:10.1002/hipo.20244 · Cited by 477
The hippocampus as a cognitive graph.
Muller, R. U., Stead, M., & Pach, J. (1996) · The Journal of general physiology · doi:10.1085/jgp.107.6.663 · Cited by 391 · cited 1×
“Pyramidal neurons in the rat hippocampus have long been known to have firing fields in localized areas of space [1,25,26].”
BackgroundA Model of Spatial Map Formation in the Hippocampus of the Rat
Blum, K. I., & Abbott, L. F. (1996) · Neural Computation · doi:10.1162/neco.1996.8.1.85 · Cited by 352
Fragmentation of grid cell maps in a multicompartment environment
Derdikman, D., Whitlock, J. R., Tsao, A., Fyhn, M., Hafting, T., Moser, M.-B., & Moser, E. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2396 · Cited by 330
On the directional firing properties of hippocampal place cells
Muller, R., Bostock, E., Taube, J., & Kubie, J. (1994) · The Journal of Neuroscience · doi:10.1523/jneurosci.14-12-07235.1994 · Cited by 322
Skaggs, W. E., & McNaughton, B. L. (1998) · The Journal of Neuroscience · doi:10.1523/jneurosci.18-20-08455.1998 · Cited by 287
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173
Self-citationStimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148
Cited by
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior
Behrens, T. E. J., Muller, T. H., Whittington, J. C. R., Mark, S., Baram, A. B., Stachenfeld, K. L., & Kurth-Nelson, Z. (2018) · Neuron · doi:10.1016/j.neuron.2018.10.002 · Cited by 1,055
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
Self-citationPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
Self-citation
Metadata
- Authors
- Gustafson, N. J., & Daw, N. D.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-10-27
- Volume
- 7
- Issue
- 10
- Pages
- e1002235
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W1973031475
- Cited by
- 61
- References
- 25
- FWCI
- 0.75
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.