Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) provides an influential characterization of the brain's mechanisms for learning to make advantageous choices. An important problem, though, is how complex tasks can be represented in a way that enables efficient learning. We consider this problem through the lens of spatial navigation, examining how two of the brain's location representations--hippocampal place cells and entorhinal grid cells--are adapted to serve as basis functions for approximating value over space for RL. Although much previous work has focused on these systems' roles in combining upstream sensory cues to track location, revisiting these representations with a focus on how they support this downstream decision function offers complementary insights into their characteristics. Rather than localization, the key problem in learning is generalization between past and present situations, which may not match perfectly. Accordingly, although neural populations collectively offer a precise representation of position, our simulations of navigational tasks verify the suggestion that RL gains efficiency from the more diffuse tuning of individual neurons, which allows learning about rewards to generalize over longer distances given fewer training experiences. However, work on generalization in RL suggests the underlying representation should respect the environment's layout. In particular, although it is often assumed that neurons track location in Euclidean coordinates (that a place cell's activity declines "as the crow flies" away from its peak), the relevant metric for value is geodesic: the distance along a path, around any obstacles. We formalize this intuition and present simulations showing how Euclidean, but not geodesic, representations can interfere with RL by generalizing inappropriately across barriers. Our proposal that place and grid responses should be modulated by geodesic distances suggests novel predictions about how obstacles should affect spatial firing fields, which provides a new viewpoint on data concerning both spatial codes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“We use a simple temporal-difference algorithm with eligibility traces [8,85] to learn weights.”
MethodMicrostructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationPath integration and the neural basis of the 'cognitive map'
McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I., & Moser, M.-B. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1932 · Cited by 2,024
Place Cells, Grid Cells, and the Brain's Spatial Representation System
Moser, E. I., Kropff, E., & Moser, M.-B. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.061307.090723 · Cited by 1,934 · cited 1×
“However, in the present work our goal is to investigate the brain’s spatial representations through the lens of their downstream computations; thus, in contrast to much work on the hippocampal system [12,35,36,40,43,44,47,48,62,63] we do not focus on the ‘‘upstream’’ computations by which the grid or place representat…”
BackgroundReverse replay of behavioural sequences in hippocampal place cells during the awake state
Foster, D. J., & Wilson, M. A. (2006) · Nature · doi:10.1038/nature04587 · Cited by 1,727 · cited 1×
“Second, across algorithms, TD-like learning mechanisms also likely interact with additional ones in the brain, and the core considerations we elucidate about efficient generalization due to appropriate state space representations crosscut these distinctions. For instance, value functions may also be updated using repl…”
BackgroundThe effects of changes in the environment on the spatial firing of hippocampal complex-spike cells
Muller, R., & Kubie, J. (1987) · The Journal of Neuroscience · doi:10.1523/jneurosci.07-07-01951.1987 · Cited by 1,581
Conjunctive Representation of Position, Direction, and Velocity in Entorhinal Cortex
Sargolini, F., Fyhn, M., Hafting, T., McNaughton, B. L., Witter, M. P., Moser, M.-B., & Moser, E. I. (2006) · Science · doi:10.1126/science.1125572 · Cited by 1,485
Spatial representation in the entorhinal cortex
Fyhn, M., Molden, S., Witter, M. P., Moser, E. I., & Moser, M.-B. (2004) · Science · doi:10.1126/science.1099901 · Cited by 1,339 · cited 1×
“Grid cells are most commonly reported in the superficial layers (II–III) of dMEC, which project directly to hippocampus [52] though they have also been reported in deep layers [53,54], where intracortical Author Summary”
BackgroundModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Cited by
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior
Behrens, T. E. J., Muller, T. H., Whittington, J. C. R., Mark, S., Baram, A. B., Stachenfeld, K. L., & Kurth-Nelson, Z. (2018) · Neuron · doi:10.1016/j.neuron.2018.10.002 · Cited by 1,055
The hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
Self-citationPredictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
Self-citation
Metadata
- Authors
- Gustafson, N. J., & Daw, N. D.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-10-27
- Volume
- 7
- Issue
- 10
- Pages
- e1002235
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W1973031475
- Cited by
- 61
- References
- 25
- FWCI
- 0.75
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning. https://app.cochat.ai/papers/10.1371/journal.pcbi.1002235
- The paper
- Gustafson, N. J., & Daw, N. D. (2011). Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1002235
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.