CoChat
Sign Up
Cited by 596Neurotransmitter Receptor Influence on BehaviorNeural and Behavioral Psychology StudiesMemory and Neural Mechanisms

Prefrontal cortex as a meta-reinforcement learning system

Open access
Wang, Jane X., Kurth-Nelson, Zeb, Kumaran, Dharshan, Tirumala, Dhruva, Soyer, Hubert, Leibo, Joel Z., Hassabis, Demis, Botvinick, Matthew·2018·Nature Neuroscience
DOI 10.1038/s41593-018-0147-8

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Over the past 20 years, neuroscience research on reward-based learning has converged on a canonical model, under which the neurotransmitter dopamine 'stamps in' associations between situations, actions and rewards by modulating the strength of synaptic connections between neurons. However, a growing number of recent findings have placed this standard model under strain. We now draw on recent advances in artificial intelligence to introduce a new theory of reward-based learning. Here, the dopamine system trains another part of the brain, the prefrontal cortex, to operate as its own free-standing learning system. This new perspective accommodates the findings that motivated the standard model, but also deals gracefully with a wider range of observations, providing a fresh foundation for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41593-018-0147-8

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Wang et al. (2018) presented observations supporting the hypothesis that prefrontal circuits may constitute a meta-reinforcement learning system.

    “For instance, Wang et al. (2018) presented observations supporting the hypothesis that prefrontal circuits may constitute a meta-reinforcement learning system.”

    From Meta-learned models of cognition (2024) · cites this paper as Wang et al. (2018)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine based on predictive hebbian learning. Journal of Neuroscience 16, 1936-1947 (1996).
  2. Glimcher, P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences 108, 15647-15654 (2011).
  3. Seo, H. & Lee, D. Cortical mechanisms for reinforcement learning in competitive games. Philosophical Transactions of the Royal Society of London B: Biological Sciences 363, 3845-3857 (2008).
  4. Tsutsui, K.-I., Grabenhorst, F., Kobayashi, S. & Schultz, W. A dynamic code for economic object valuation in prefrontal cortex neurons. Nature Communications 7, 12554 (2016).
  5. Bromberg-Martin, E.S., Matsumoto, M., Hong, S. & Hikosaka, O. A pallidus-habenula- dopamine pathway signals inferred stimulus values. Journal of neurophysiology 104, 1068-1076 (2010).
  6. Sadacca, B.F., Jones, J.L. & Schoenbaum, G. Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework. Elife 5, e13665 (2016).
  7. O'Reilly, R.C. & Frank, M.J. Making working memory work: a computational model of learning in prefrontal cortex and basal ganglia. Neural Computation 18, 283-328 (2006).
  8. Song, H.F., Yang, G.R. & Wang, X.-J. Reward-based training of recurrent neural networks for cognitive and value-based tasks. eLife 6, e21492 (2017).
  9. Enel, P., Procyk, E., Quilodran, R. & Dominey, P.F. Reservoir computing properties of neural dynamics in prefrontal cortex. PLoS Comput Biol 12, e1004967 (2016).
  10. Redish, A.D., Jensen, S., Johnson, A. & Kurth-Nelson, Z. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychological review 114, 784 (2007).
  11. Kennerley, S.W. & Wallis, J.D. Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables. European Journal of Neuroscience 29, 2061-2073 (2009).
  12. Houk, J.C., Adams, C.M. & Barto, A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. in Models of Information Processing in the Basal Ganglia (ed. J.C. Houk & D.G. Davis) 249-270 (MIT Press, Cambridge, 1995).
  13. Rougier, N.P., Noell, D.C., Braver, T.S., Cohen, J.D. & O'Reilly, R.C. Prefrontal cortex and flexible cognitive control: Rules without symbols. Proceedings of the National Academy of Sciences 102, 7338-7343 (2005).
  14. Acuna, D. & Schrater, P.R. Structure learning in human sequential decision-making. in Advances in Neural Information Processing Systems 1-8 (2009).
  15. Collins, A.G. & Frank, M.J. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. European Journal of Neuroscience 35, 1024-1035 (2012).
  16. Thrun, S. & Pratt, L. Learning to learn (Springer Science & Business Media, 2012).
  17. Duan, Y., et al. RL squared: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv preprint arXiv:1611.02779 (2016). · arXiv
  18. Mnih, V., et al. Asynchronous methods for deep reinforcement learning. in International Conference on Machine Learning (2016).
  19. Baum, W.M. Optimization and the matching law as accounts of instrumental behavior. Journal of the experimental analysis of behavior 36, 387-403 (1981).
  20. Iigaya, K. Adaptive learning and decision-making under uncertainty by metaplastic synapses guided by a surprise detection system. Elife 5, e18073 (2016).
  21. Nassar, M.R., Wilson, R.C., Heasly, B. & Gold, J.I. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. Journal of Neuroscience 30, 12366-12378 (2010).
  22. Chan, S.C., NIv, Y. & Norman, K.A. A Probability Distribution over Latent Causes, in the Orbitofrontal Cortex. Journal of Neuroscience 36, 7817-7828 (2016).
  23. Hampton, A.N., Bossaerts, P. & O'doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience 26, 8360-8367 (2006).
  24. Akam, T., Costa, R. & Dayan, P. Simple plans or sophisticated habits? State, transition and learning interactions in the two-step task. PLoS Comput Biol 11, e1004648 (2015).
  25. Harlow, H.F. The formation of learning sets. Psychological review 56, 51 (1949).
  26. Kim, K.M., et al. Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement. PloS one 7, e33612 (2012).
  27. Kim, H., Lee, D. & Jung, M.W. Signals for previous goal choice persist in the dorsomedial, but not dorsolateral striatum of rats. Journal of Neuroscience 33, 52-63 (2013).
  28. Gershman, S.J. & Daw, N.D. Reinforcement learning and episodic memory in humans and animals: An integrative framework. Annual review of psychology 68, 101-128 (2017).
  29. Graves, A., Jaitly, N. & Mohamed, A.-r. Hybrid speech recognition with deep bidirectional LSTM. in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on 273- 278 (IEEE, 2013).
  30. Leibo, J. Z., d'Autume, C. D. M., Zoran, D., Amos, D., Beattie, C., Anderson, K., et al. (2018). Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents. arXiv preprint arXiv:1801.08116. · arXiv
  31. Deng, J., et al. Imagenet: A large-scale hierarchical image database. in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on 248-255 (IEEE, 2009).

Cited by

Sorted by citations

Authors
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M.
Affiliations
Google DeepMind (United Kingdom) (GB)
Corresponding author
Botvinick, Matthew
ORCID
Venue
Nature Neuroscience
Publisher
Springer Science and Business Media LLC
Published
2018-05-14
Volume
21
Issue
6
Pages
860-868
ISSN
  • 1097-6256 (ISSN-L)
  • 1546-1726
DOI
10.1038/s41593-018-0147-8
OpenAlex
W2949369413
Cited by
596
References
9
Citations by year
2026: 59 · 2025: 69 · 2024: 72 · 2023: 61 · 2022: 80 · 2021: 106 · 2020: 79 · 2019: 50 · 2018: 19
FWCI
28.93
Type
journal article
Language
en
Open access
Licence
www.springer.com/tdm (tdm)
Locations
  • Nature Neuroscience (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed (publishedVersion)
  • Max Planck Digital Library (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
Topics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
Keywords
prefrontal cortex, dopamine, reward learning, meta-learning, reinforcement learning, synaptic plasticity
MeSH terms
Algorithms, Animals, Artificial Intelligence, Computer Simulation, Dopamine, Humans, Learning, Models, Neurological, Reinforcement, Psychology, Reward, Prefrontal Cortex, Optogenetics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review