Metalearning and neuromodulation
Abstract
This paper presents a computational theory on the roles of the ascending neuromodulatory systems from the viewpoint that they mediate the global signals that regulate the distributed learning mechanisms in the brain. Based on the review of experimental data and theoretical models, it is proposed that dopamine signals the error in reward prediction, serotonin controls the time scale of reward prediction, noradrenaline controls the randomness in action selection, and acetylcholine controls the speed of memory update. The possible interactions between those neuromodulators and the environment are predicted on the basis of computational theory of metalearning.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The perspective on optimum policies as prior expectations may call for a reappraisal of reinforcement learning as an explanation for behavior and the role of dopamine in coding unpredicted reward.
“The perspective on optimum policies as prior expectations may call for a reappraisal of reinforcement learning as an explanation for behavior and, in particular, the role of dopamine in coding unpredicted reward (Montague et al. 1995;Schultz et al. 1997;Doya 2002)”
From Action and behavior: a free-energy formulation (2010) · cites this paper as Doya 2002
Cites
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · see citing works
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190 · see citing works
Complementary roles of basal ganglia and cerebellum in learning and motor control
Doya, K. (2000) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00153-7 · Cited by 964 · see citing works
Self-citationA cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879 · see citing works
Reward expectancy in primate prefrental neurons
Watanabe, M. (1996) · Nature · doi:10.1038/382629a0 · Cited by 684 · cited 1× · see citing works
“V(s(t)) V(s(t-1)) are dependent on the amount and type of the reward associated with the memorized items (Watanabe, 1996; Tremblay & Schultz, 2000).”
BackgroundLocus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task
Aston-Jones, G., Rajkowski, J., Kubiak, P., & Alexinsky, T. (1994) · The Journal of Neuroscience · doi:10.1523/jneurosci.14-07-04467.1994 · Cited by 637 · see citing works
The Physiological Role of 5-HT<sub>2A</sub>Receptors in Working Memory
Williams, G. V., Rao, S. G., & Goldman-Rakic, P. S. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-07-02843.2002 · Cited by 311 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · see citing works
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224 · see citing works
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · see citing works
Inhibition and impulsivity: behavioral and neural basis of response control
Bari, A., & Robbins, T. W. (2013) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2013.06.005 · Cited by 1,921 · see citing works
The free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831 · see citing works
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · see citing works
Habits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · see citing works
Social reward requires coordinated activity of nucleus accumbens oxytocin and serotonin
Dölen, G., Darvishzadeh, A., Huang, K. W., & Malenka, R. C. (2013) · Nature · doi:10.1038/nature12518 · Cited by 1,161 · see citing works
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966 · see citing works
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844 · see citing works
Action and behavior: a free-energy formulation
Friston, K. J., Daunizeau, J., Kilner, J., & Kiebel, S. J. (2010) · Biological Cybernetics · doi:10.1007/s00422-010-0364-z · Cited by 866 · see citing works
“may call for a reappraisal of reinforcement learning as an explanation for behavior and, in particular, the role of dopamine in coding unpredicted reward (Montague et al. 1995; Schultz et al. 1997; Doya 2002 ): If dopamine encodes precision through its classical neuromodulatory effects (cf., Yu and Dayan 2005) how can…”
BackgroundHuman insula activation reflects risk prediction errors as well as risk
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4286-07.2008 · Cited by 784 · see citing works
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · see citing works
Self-citationDoya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642 · see citing works
Self-citationActivity of striatal neurons reflects dynamic encoding and recoding of procedural memories
Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z., & Graybiel, A. M. (2005) · Nature · doi:10.1038/nature04053 · Cited by 595 · see citing works
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580 · see citing works
Representation of negative motivational value in the primate lateral habenula
Matsumoto, M., & Hikosaka, O. (2009) · Nature Neuroscience · doi:10.1038/nn.2233 · Cited by 547 · see citing works
Reasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making
Collins, A., & Koechlin, E. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001293 · Cited by 558 · see citing works
“The actor selective mapping then determines the response to stimulus using a softmax policy (inverse temperature b and noise e) [11,15,28].”
MethodSerotonin and dopamine: unifying affective, activational, and decision functions.
Cools, R., Nakamura, K., & Daw, N. D. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.121 · Cited by 464 · see citing works
Reinforcement learning or active inference?
Friston, K. J., Daunizeau, J., & Kiebel, S. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006421 · Cited by 443 · see citing works
Central Cholinergic Neurons Are Rapidly Recruited by Reinforcement Feedback
Hangya, B., Ranade, S. P., Lorenc, M., & Kepecs, A. (2015) · Cell · doi:10.1016/j.cell.2015.07.057 · Cited by 399 · see citing works
Dopamine, affordance and active inference
Friston, K. J., Shiner, T., FitzGerald, T., Galea, J. M., Adams, R., Brown, H., Dolan, R. J., Moran, R., Stephan, K. E., & Bestmann, S. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002327 · Cited by 381 · see citing works
Confidence as Bayesian probability: From neural origins to behavior
Meyniel, F., Sigman, M., & Mainen, Z. F. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.039 · Cited by 349 · see citing works
Serotonin in Affective Control
Dayan, P., & Huys, Q. J. M. (2009) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135607 · Cited by 381 · see citing works
Hierarchical Prediction Errors in Midbrain and Basal Forebrain during Sensory Learning
Iglesias, S., Mathys, C., Brodersen, K. H., Kasper, L., Piccirelli, M., den Ouden, H. E. M., & Stephan, K. E. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.009 · Cited by 342 · see citing works
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346 · see citing works
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277 · see citing works
Uncertainty, epistemics and active inference
Parr, T., & Friston, K. J. (2017) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0376 · Cited by 252 · see citing works
Lateral habenula neurons signal errors in the prediction of reward information
Bromberg-Martin, E. S., & Hikosaka, O. (2011) · Nature Neuroscience · doi:10.1038/nn.2902 · Cited by 275 · see citing works
Meta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261 · see citing works
Self-citationHu, H. (2016) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-070815-014106 · Cited by 253 · see citing works
Serotonin Selectively Modulates Reward Value in Human Decision-Making
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P., & Dolan, R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.0053-12.2012 · Cited by 252 · see citing works
Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems
Ogawa, S. K., Cohen, J. Y., Hwang, D., Uchida, N., & Watabe-Uchida, M. (2014) · Cell Reports · doi:10.1016/j.celrep.2014.06.042 · Cited by 249 · see citing works
Computational Models of Performance Monitoring and Cognitive Control
Alexander, W. H., & Brown, J. W. (2010) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2010.01085.x · Cited by 193 · see citing works
Twenty-Five Lessons from Computational Neuromodulation
Dayan, P. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.027 · Cited by 190 · see citing works
Serotonin neurons modulate learning rate through uncertainty
Grossman, C. D., Bari, B. A., & Cohen, J. Y. (2022) · Current Biology · doi:10.1016/j.cub.2021.12.006 · Cited by 164 · see citing works
Skvortsova, V., Palminteri, S., & Pessiglione, M. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1350-14.2014 · Cited by 177 · see citing works
Meta-learning in natural and artificial intelligence
Wang, J. X. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.01.002 · Cited by 171 · see citing works
The role of serotonin in the regulation of patience and impulsivity
Miyazaki, K., Miyazaki, K. W., & Doya, K. (2012) · Molecular Neurobiology · doi:10.1007/s12035-012-8232-6 · Cited by 162 · see citing works
Self-citationSerotonin and decision making processes
Homberg, J. R. (2012) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2011.06.001 · Cited by 167 · see citing works
Emotion in reinforcement learning agents and robots: a survey
Moerland, T. M., Broekens, J., & Jonker, C. M. (2018) · Machine Learning · doi:10.1007/s10994-017-5666-0 · Cited by 205 · see citing works
Brain networks for confidence weighting and hierarchical inference during probabilistic learning
Meyniel, F., & Dehaene, S. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1615773114 · Cited by 138 · see citing works
Bayesian modelling of Jumping-to-Conclusions bias in delusional patients
Moutoussis, M., Bentall, R. P., El-Deredy, W., & Dayan, P. (2011) · Cognitive Neuropsychiatry · doi:10.1080/13546805.2010.548678 · Cited by 137 · see citing works
<scp>O</scp>xytocin modulation of neural circuits for social behavior
Marlin, B. J., & Froemke, R. C. (2017) · Developmental Neurobiology · doi:10.1002/dneu.22452 · Cited by 130 · see citing works
Griffiths, K. R., Morris, R. W., & Balleine, B. W. (2014) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2014.00101 · Cited by 118 · see citing works
Iigaya, K., Fonseca, M. S., Murakami, M., Mainen, Z. F., & Dayan, P. (2018) · Nature Communications · doi:10.1038/s41467-018-04840-2 · Cited by 120 · see citing works
A Genetically Defined Compartmentalized Striatal Direct Pathway for Negative Reinforcement
Xiao, X., Deng, H., Furlan, A., Yang, T., Zhang, X., Hwang, G.-R., Tucciarone, J., Wu, P., He, M., Palaniswamy, R., et al. (2020) · Cell · doi:10.1016/j.cell.2020.08.032 · Cited by 101 · see citing works
Reinforcement learning: Computational theory and biological mechanisms
Doya, K. (2007) · HFSP journal · doi:10.2976/1.2732246 · Cited by 150 · see citing works
Self-citationThe successor representation and temporal context
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012) · Neural Computation · doi:10.1162/neco_a_00282 · Cited by 117 · see citing works
Activation of dorsal raphe serotonin neurons is necessary for waiting for delayed rewards
Miyazaki, K. W., Miyazaki, K., & Doya, K. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.0915-12.2012 · Cited by 106 · see citing works
Self-citationPalminteri, S., Lebreton, M., Worbe, Y., Hartmann, A., Lehéricy, S., Vidailhet, M., Grabli, D., & Pessiglione, M. (2011) · Brain · doi:10.1093/brain/awr147 · Cited by 90 · see citing works
Vinckier, F., Gaillard, R., Palminteri, S., Rigoux, L., Salvador, A., Fornito, A., Adapa, R., Krebs, M. O., Pessiglione, M., & Fletcher, P. C. (2016) · Molecular Psychiatry · doi:10.1038/mp.2015.73 · Cited by 87 · see citing works
Neural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X.-J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.044 · Cited by 77 · see citing works
Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty
Sandhu, T. R., Xiao, B., & Lawson, R. P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105123 · Cited by 51 · see citing works
Jepma (2010) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2010.00170 · Cited by 45 · see citing works
Dopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · see citing works
Noradrenergic and Cholinergic Modulation of Belief Updating
Jepma, M., Brown, S. B. R. E., Murphy, P. R., Koelewijn, S. C., de Vries, B., van den Maagdenberg, A. M., & Nieuwenhuis, S. (2018) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01317 · Cited by 39 · see citing works
Meta-learned models of cognition
Binz, M., Dasgupta, I., Jagadish, A. K., Botvinick, M., Wang, J. X., & Schulz, E. (2024) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23003266 · Cited by 38 · see citing works
- Authors
- Doya, K.
- Affiliations
- Advanced Telecommunications Research Institute International (JP)
- Corresponding author
- Doya, Kenji
- ORCID
- Doya, Kenji
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2002-06
- Volume
- 15
- Issue
- 4-6
- Pages
- 495-506
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2161563886
- Cited by
- 704
- References
- 7
- Citations by year
- 2026: 24 · 2025: 32 · 2024: 30 · 2023: 32 · 2022: 26 · 2021: 35 · 2020: 34 · 2019: 23 · 2018: 30 · 2017: 26 · 2016: 31 · 2015: 30 · 2014: 42 · 2013: 44 · 2012: 34
- FWCI
- 5.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- meta-learning, neuromodulation, dopamine, serotonin, norepinephrine, acetylcholine, reward prediction error, action selection, distributed learning
- MeSH terms
- Algorithms, Animals, Brain, Humans, Learning, Models, Biological, Neurotransmitter Agents
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
