CoChat

Linear reinforcement learning in planning, grid fields, and cognitive control

Open access
Piray, Payam, Daw, Nathaniel D. · 2021 · Nature Communications

Published version of a preprint

Cited by 78

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Abstract

It is thought that the brain’s judicious reuse of previous computation underlies our ability to plan flexibly, but also that inappropriate reuse gives rise to inflexibilities like habits and compulsion. Yet we lack a complete, realistic account of either. Building on control engineering, here we introduce a model for decision making in the brain that reuses a temporally abstracted map of future events to enable biologically-realistic, flexible choice at the expense of specific, quantifiable biases. It replaces the classic nonlinear, model-based optimization with a linear approximation that softly maximizes around (and is weakly biased toward) a default policy. This solution demonstrates connections between seemingly disparate phenomena across behavioral neuroscience, notably flexible replanning with biases and cognitive control. It also provides insight into how the brain can represent maps of long-distance contingencies stably and componentially, as in entorhinal response fields, and exploit them to guide choice even under changing goals.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-021-25123-3

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 51

  1. Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J. & Daw, N. D. Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput. Biol. 13, e1005768 (2017). · DOI · Paper page
  2. Momennejad, I. et al. The successor representation in human reinforcement learning. Nat. Hum. Behav. 1, 680–692 (2017). · DOI · Paper page
  3. Bellman, R. E. Dynamic Programming (Princeton University Press, 1957).
  4. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 2018).
  5. Dickinson, A. & Balleine, B. W. In Steven’s Handbook of Experimental Psychology: Learning, Motivation, and Emotion Vol. 3 (ed. Gallistel, C. R.) 497–533 (Wiley, 2002).
  6. Wimmer, G. E. & Shohamy, D. Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338, 270–273 (2012). · DOI · Paper page
  7. Behrens, T. E. J. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018). · DOI · Paper page
  8. Keramati, M., Smittenaar, P., Dolan, R. J. & Dayan, P. Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum. Proc. Natl Acad. Sci. USA 113, 12868–12873 (2016). · DOI · Paper page
  9. Huys, Q. J. M. et al. Interplay of approximate planning strategies. Proc. Natl Acad. Sci. USA 112, 3098–3103 (2015). · DOI · Paper page
  10. Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624 (1993). · DOI · Paper page
  11. Lehnert, L., Tellex, S. & Littman, M. L. Advantages and limitations of using successor features for transfer in reinforcement learning. Preprint at https://arxiv.org/abs/1708.00102 (2017). · arXiv
  12. Botvinick, M. M., Niv, Y. & Barto, A. C. Hierarchically organized behavior and its neural foundations: a reinforcement-learning perspective. Cognition 113, 262–280 (2009). · DOI · Paper page
  13. Kappen, H. J. Linear theory for control of nonlinear stochastic systems. Phys. Rev. Lett. 95, 200201 (2005). · DOI · Paper page
  14. Todorov, E. In Advances in Neural Information Processing Systems 19 (eds. Schölkopf, B. et al.) 1369–1376 (MIT Press, 2007). · DOI · Paper page
  15. Todorov, E. Efficient computation of optimal actions. Proc. Natl Acad. Sci. USA 106, 11478–11483 (2009). · DOI · Paper page
  16. Gustafson, N. J. & Daw, N. D. Grid cells, place cells, and geodesic generalization for spatial reinforcement learning. PLOS Comput. Biol. 7, e1002235 (2011). · DOI · Paper page
  17. Mahadevan, S. Representation policy iteration. Preprint at https://arxiv.org/abs/1207.1408 (2012). · arXiv
  18. Mahadevan, S. & Maggioni, M. Proto-value functions: a Laplacian framework for learning representation and control in Markov decision processes. J. Mach. Learn. Res. 8, 2169–2231 (2007).
  19. Carpenter, F., Manson, D., Jeffery, K., Burgess, N. & Barry, C. Grid cells form a global representation of connected environments. Curr. Biol. 25, 1176–1182 (2015). · DOI · Paper page
  20. Sanguinetti-Scheck, J. I. & Brecht, M. Home, head direction stability, and grid cell distortion. J. Neurophysiol. 123, 1392–1406 (2020). · DOI
  21. Stroop, J. R. Studies of interference in serial verbal reactions. J. Exp. Psychol. 18, 643–662 (1935). · DOI · Paper page
  22. Cohen, J. D., Dunbar, K. & McClelland, J. L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990). · DOI · Paper page
  23. Estes, W. K. A Discriminative conditioning. I. A discriminative property of conditioned anticipation. J. Exp. Psychol. 32, 150–155 https://doi.org/10.1037/h0058316 (1943). · DOI · Paper page
  24. Corbit, L. H. & Balleine, B. W. In Behavioral Neuroscience of Motivation (eds. Simpson, E. H. & Balsam, P. D.) 259–289 (Springer International Publishing, 2016). · DOI
  25. Tolman, E. C. & Gleitman, H. Studies in learning and motivation; equal reinforcements in both end-boxes; followed by shock in one end-box. J. Exp. Psychol. 39, 810–819 (1949). · DOI
  26. Sutton, R. S. TD models: modeling the world at a mixture of time scales. In Machine Learning Proceedings 1995 (eds. Prieditis, A. & Russell, S.) 531–539 (Morgan Kaufmann, 1995). · DOI
  27. Baram, A. B., Muller, T. H., Whittington, J. C. R. & Behrens, T. E. J. Intuitive planning: global navigation through cognitive maps based on grid-like codes. Preprint at bioRxiv https://doi.org/10.1101/421461 (2018). · DOI
  28. Solstad, T., Boccara, C. N., Kropff, E., Moser, M.-B. & Moser, E. I. Representation of geometric borders in the entorhinal cortex. Science 322, 1865–1868 (2008). · DOI · Paper page
  29. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P. & Dolan, R. J. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215 (2011). · DOI · Paper page
  30. Botvinick, M. M. & Cohen, J. D. The computational and neural basis of cognitive control: charted territory and new frontiers. Cogn. Sci. 38, 1249–1285 (2014). · DOI · Paper page
  31. Westbrook, A., Kester, D. & Braver, T. S. What is the subjective cost of cognitive effort? load, trait, and aging effects revealed by economic preference. PLoS ONE 8, e68210 (2013). · DOI · Paper page
  32. Krebs, R. M., Boehler, C. N. & Woldorff, M. G. The influence of reward associations on conflict processing in the Stroop task. Cognition 117, 341–347 (2010). · DOI · Paper page
  33. Dayan, P., Niv, Y., Seymour, B. & Daw, N. D. The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153–1160 (2006). · DOI · Paper page
  34. Afsardeir, A. & Keramati, M. Behavioural signatures of backward planning in animals. Eur. J. Neurosci. 47, 479–487 (2018). · DOI
  35. aarnoja, T., Tang, H., Abbeel, P. & Levine, S. Reinforcement learning with deep energy-based policies. In Proceedings 34th International Conference on Machine Learning 70 1352–1361 (JMLR.org, 2017).
  36. Cushman, F. & Morris, A. Habitual control of goal selection in humans. Proc. Natl. Acad. Sci. USA 112, 13817–13822 (2015). · DOI · Paper page
  37. Sutton, R. S. & Pinette, B. The learning of world models by connectionist networks. In Seventh Annual Conference of the Cognitive Science Society 54–64 (1985).
  38. Jang, J.-S., Lee, S.-Y. & Shin, S.-Y. An optimization network for matrix inversion. In NIPS Proc. 397–401 (MIT Press, 1987).
  39. Dordek, Y., Soudry, D., Meir, R. & Derdikman, D. Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis. eLife 5, e10094 (2016). · DOI · Paper page
  40. Barry, C., Hayman, R., Burgess, N. & Jeffery, K. J. Experience-dependent rescaling of entorhinal grids. Nat. Neurosci. 10, 682–684 (2007). · DOI · Paper page
  41. Pérez-Escobar, J. A., Kornienko, O., Latuske, P., Kohler, L. & Allen, K. Visual landmarks sharpen grid cell metric and confer context specificity to neurons of the medial entorhinal cortex. eLife 5, e16937 (2016). · DOI · Paper page
  42. Ismakov, R., Barak, O., Jeffery, K. & Derdikman, D. Grid cells encode local positional information. Curr. Biol. 27, 2337–2343.e3 (2017). · DOI · Paper page
  43. Gershman, S. & Wilson, R. The neural costs of optimal control. In Advances in Neural Information Processing Systems Vol. 23 (eds. Lafferty, J. D. et al.) 712–720 (Curran Associates, Inc., 2010).
  44. Woodford, M. Prospect theory as efficient perceptual distortion. Am. Econ. Rev. 102, 41–46 (2012). · DOI
  45. Ziebart, B. D., Maas, A., Bagnell, J. A. & Dey, A. K. Maximum entropy inverse reinforcement learning. In Proc. 23rd National Conference on Artificial intelligence Vol. 3, 1433–1438 (AAAI Press, 2008).
  46. Botvinick, M. & Toussaint, M. Planning as inference. Trends Cogn. Sci. 16, 485–488 (2012). · DOI · Paper page
  47. Barto, A. G. In Models of Information Processing in the Basal Ganglia (eds. Houk, J. C. et al.) 215–232 (MIT Press, 1995). · DOI · Paper page
  48. Teh, Y. et al. In Advances in Neural Information Processing Systems Vol. 30 (eds. Guyon, I. et al.) 4496–4506 (Curran Associates, Inc., 2017).
  49. Zorowitz, S., Momennejad, I. & Daw, N. D. Anxiety, avoidance, and sequential evaluation. Comput. Psychiatr. 4 (2020). · DOI · Paper page
  50. Huys, Q. J. M. & Renz, D. A formal valuation framework for emotions and their control. Biol. Psychiatry 82, 413–420 (2017). · DOI
  51. Piray, P., Toni, I. & Cools, R. Human choice strategy varies with anatomical projections from ventromedial prefrontal cortex to medial striatum. J. Neurosci. 36, 2857–2867 (2016). · DOI

Cited by

See all 78

Metadata

Authors
Piray, P., & Daw, N. D.
Venue
Nature Communications
Published
2021-08-16
Volume
12
Issue
1
Pages
4942-4942
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3195433036
Cited by
78
References
33
FWCI
5.12
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Linear reinforcement learning in planning, grid fields, and cognitive control. https://app.cochat.ai/papers/10.1038/s41467-021-25123-3
The paper
Piray, P., & Daw, N. D. (2021). Linear reinforcement learning in planning, grid fields, and cognitive control. Nature Communications. https://doi.org/10.1038/s41467-021-25123-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.