Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Knowledge of the structure of a problem, such as relationships between stimuli, enables rapid learning and flexible inference. Humans and other animals can abstract this structural knowledge and generalize it to solve new problems. For example, in spatial reasoning, shortest-path inferences are immediate in new environments. Spatial structural transfer is mediated by cells in entorhinal and (in humans) medial prefrontal cortices, which maintain their co-activation structure across different environments and behavioral states. Here, using fMRI, we show that entorhinal and ventromedial prefrontal cortex (vmPFC) representations perform a much broader role in generalizing the structure of problems. We introduce a task-remapping paradigm, where subjects solve multiple reinforcement learning (RL) problems differing in structural or sensory properties. We show that, as with space, entorhinal representations are preserved across different RL problems only if task structure is preserved. In vmPFC and ventral striatum, representations of prediction error also depend on task structure.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson, B. C., Blacker, D., Buckner, R. L., Dale, A. M., Maguire, R. P., Hyman, B. T., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.021 · Cited by 13,324
Jenkinson, M., Bannister, P., Brady, M., & Smith, S. (2002) · NeuroImage · doi:10.1006/nimg.2002.1132 · Cited by 11,888
Fischl, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.021 · Cited by 9,526
Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., et al. (2002) · Neuron · doi:10.1016/s0896-6273(02)00569-x · Cited by 8,610
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Nonparametric permutation tests for functional neuroimaging: A primer with examples
Nichols, T. E., & Holmes, A. P. (2002) · Human Brain Mapping · doi:10.1002/hbm.1058 · Cited by 6,256
The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
Cortical Surface-Based Analysis
Fischl, B., Sereno, M. I., & Dale, A. M. (1999) · NeuroImage · doi:10.1006/nimg.1998.0396 · Cited by 6,147
Microstructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563
Representational similarity analysis-connecting the branches of systems neuroscience
Kriegeskorte, N. (2008) · Frontiers in Systems Neuroscience · doi:10.3389/neuro.06.004.2008 · Cited by 3,355
The formation of learning sets.
Harlow, H. F. (1949) · Psychological Review · doi:10.1037/h0062474 · Cited by 1,959
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Tracking the Hemodynamic Responses to Reward and Punishment in the Striatum
Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000) · Journal of Neurophysiology · doi:10.1152/jn.2000.84.6.3072 · Cited by 1,116
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
Self-citationThe hippocampus as a predictive map
Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4650 · Cited by 1,000
Distributed Neural Representation of Expected Value
Knutson, B., Taylor, J., Kaufman, M., Peterson, R., & Glover, G. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.0642-05.2005 · Cited by 955
Organizing conceptual knowledge in humans with a gridlike code
Constantinescu, A. O., O’Reilly, J. X., & Behrens, T. E. J. (2016) · Science · doi:10.1126/science.aaf0941 · Cited by 937
Self-citationOrbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885
Evidence for grid cells in a human memory network
Doeller, C. F., Barry, C., & Burgess, N. (2010) · Nature · doi:10.1038/nature08704 · Cited by 828
Boundary Vector Cells in the Subiculum of the Hippocampal Formation
Lever, C., Burton, S., Jeewajee, A., O'Keefe, J., & Burgess, N. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1319-09.2009 · Cited by 753
Hippocampal remapping and grid realignment in entorhinal cortex
Fyhn, M., Hafting, T., Treves, A., Moser, M.-B., & Moser, E. I. (2007) · Nature · doi:10.1038/nature05601 · Cited by 749
How does nature program neuron types?
Borst, A. (2008) · Frontiers in Neuroscience · doi:10.3389/neuro.01.016.2008 · Cited by 705
Homo naledi, a new species of the genus Homo from the Dinaledi chamber. South Africa.
Berger, L. R., Hawks, J., de Ruiter, D. J., Churchill, S. E., Schmid, P., Delezene, L. K., Kivell, T. L., Garvin, H. M., Williams, S. A., DeSilva, J. M., et al. (2015) · eLife · doi:10.7554/elife.09560 · Cited by 703
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660
Self-citationSchema-dependent gene activation and memory encoding in neocortex
Tse, D., Takeuchi, T., Kakeyama, M., Kajii, Y., Okuno, H., Tohyama, C., Bito, H., & Morris, R. G. M. (2011) · Science · doi:10.1126/science.1205274 · Cited by 618
Prefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Walton, M. E., Behrens, T. E. J., Buckley, M. J., Rudebeck, P. H., & Rushworth, M. F. S. (2010) · Neuron · doi:10.1016/j.neuron.2010.02.027 · Cited by 397
Self-citationA map of abstract relational knowledge in the human hippocampal-entorhinal cortex
Garvert, M. M., Dolan, R. J., & Behrens, T. E. (2017) · eLife · doi:10.7554/elife.17086 · Cited by 379
Self-citationObject-vector coding in the medial entorhinal cortex
Høydal, Ø. A., Skytøen, E. R., Andersson, S. O., Moser, M.-B., & Moser, E. I. (2019) · Nature · doi:10.1038/s41586-019-1077-7 · Cited by 366
Influence of boundary removal on the spatial representations of the medial entorhinal cortex
Savelli, F., Yoganarasimha, D., & Knierim, J. J. (2008) · Hippocampus · doi:10.1002/hipo.20511 · Cited by 363
Learning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326
The entorhinal cognitive map is attracted to goals
Boccara, C. N., Nardin, M., Stella, F., O’Neill, J., & Csicsvari, J. (2019) · Science · doi:10.1126/science.aav4837 · Cited by 288
Remembered reward locations restructure entorhinal spatial maps
Butler, W. N., Hardcastle, K., & Giocomo, L. M. (2019) · Science · doi:10.1126/science.aav5297 · Cited by 232
Grid-like Neural Representations Support Olfactory Navigation of a Two-Dimensional Odor Space
Bao, X., Gjorgieva, E., Shanahan, L. K., Howard, J. D., Kahnt, T., & Gottfried, J. A. (2019) · Neuron · doi:10.1016/j.neuron.2019.03.034 · Cited by 207
Rudebeck, P. H., Saunders, R. C., Lundgren, D. A., & Murray, E. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.07.042 · Cited by 170
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151
Generalization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111
Trettel, S. G., Trimper, J. B., Hwaun, E., Fiete, I. R., & Colgin, L. L. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0359-6 · Cited by 108
Reward-Guided Learning with and without Causal Attribution
Jocham, G., Brodersen, K. H., Constantinescu, A. O., Kahn, M. C., Ianni, A. M., Walton, M. E., Rushworth, M. F. S., & Behrens, T. E. J. (2016) · Neuron · doi:10.1016/j.neuron.2016.02.018 · Cited by 92
Self-citationThe Emergence of Knowledge and How it Supports the Memory for Novel Related Information
Sommer, T. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhw031 · Cited by 91
Cited by
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660
Self-citationIf deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397
Map Making: Constructing, Combining, and Inferring on Abstract Cognitive Maps
Park, S. A., Miller, D. S., Nili, H., Ranganath, C., & Boorman, E. D. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.030 · Cited by 270
Self-citationMedial and orbital frontal cortex in decision-making and flexible behavior
Klein-Flügge, M. C., Bongioanni, A., & Rushworth, M. F. S. (2022) · Neuron · doi:10.1016/j.neuron.2022.05.022 · Cited by 244
Is the core function of orbitofrontal cortex to signal values or make predictions?
Zhou, J., Gardner, M. P., & Schoenbaum, G. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.02.011 · Cited by 57
Metadata
- Authors
- Baram, A. B., Muller, T. H., Nili, H., Garvert, M. M., & Behrens, T. E. J.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2021-02
- Volume
- 109
- Issue
- 4
- Pages
- 713-723.e7
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W3113840761
- Cited by
- 140
- References
- 42
- FWCI
- 7.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.