The DOI matches its Crossref record; 16 of the 17 references it cites were found in a registry or an index.
Meta-learning in Reinforcement Learning
Abstract
Meta-parameters in reinforcement learning should be tuned to the environmental dynamics and the animal performance. Here, we propose a biologically plausible meta-reinforcement learning algorithm for tuning these meta-parameters in a dynamic, adaptive manner. We tested our algorithm in both a simulation of a Markov decision task and in a non-linear control task. Our results show that the algorithm robustly finds appropriate meta-parameter values, and controls the meta-parameter time course, in both static and dynamic environments. We suggest that the phasic and tonic components of dopamine neuron firing can encode the signal required for meta-learning of reinforcement learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564
Aston-Jones, G., & Bloom, F. (1981) · The Journal of Neuroscience · doi:10.1523/jneurosci.01-08-00876.1981 · Cited by 1,960
An opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 995
Self-citationOpponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844
Metalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704
Self-citation5-HT and motor control: a hypothesis
Jacobs, B. L., & Fornal, C. A. (1993) · Trends in Neurosciences · doi:10.1016/0166-2236(93)90090-9 · Cited by 452
On the Complexity of Solving Markov Decision ProblemsPreprint
Littman, M. L., Dean, T., & Kaelbling, L. P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4971 · Cited by 434
A stochastic reinforcement learning algorithm for learning real-valued functions
Gullapalli, V. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90056-q · Cited by 292
The role of exploration in learning control
Thrun, S. (1992) · Cited by 226
Research works printed but not linked: 1
- Tanaka, 2002, Functional MRI study of short-term and long-term prediction of reward, Proceedings of the Eighth International Conference on Functional Mapping of the Human Brain, Sendai, Japan, 1062
Cited by
Sorted by citations
Items 201–260 of 260
Task Scheduling for Single Satellite Observation Based on Meta-Reinforcement Learning
Li, Z., Li, Z., & Bai, Y. (2025) · Journal of Aerospace Information Systems · doi:10.2514/1.i011619 · Cited by 1
An improved transfer learning for surface defect detection in edge-cloud continuum environments
Tang, B. T., Lin, Y., Cao, Y., Zhang, L., Ma, X., & Yang, Q. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-025-07993-2 · Cited by 1
“Schweighofer et al. [23] proposed a biologically plausible meta-reinforce-ment learning algorithm to adjust the meta-parameters in reinforcement learning in a dynamic and adaptive manner.”
Yasuda, T., Motohiro, W. A. D. A., & Ohkura, K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23232-9_15 · Cited by 1
“To overcome this problem, the present paper proposes another extended BRL that focusses on meta-parameters that modulate the learning and are of crucial importance for RL. In this study, meta-learning [14, 15 ], which is the capability of the learning algorithm to dynamically adjust its meta-parameters, is employed.”
MethodAttentive One-Shot Meta-Imitation Learning From Visual Demonstration
Bhutani, V., Majumder, A., Vankadari, M. B., Dutta, S., Asati, A., & Kumar, S. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812281 · Cited by 1
Plaat, A. (2022) · Springer eBooks · doi:10.1007/978-981-19-0638-1_9 · Cited by 1
Efficient Data-Driven Modeling of Nonlinear Dynamical Systems via Metalearning
Li, S., & Yang, Y. (2023) · Journal of Engineering Mechanics · doi:10.1061/jenmdt.emeng-6917 · Cited by 1
Few-shot Edge Classification in Graph Meta-learning
Yang, X., & Xu, J. (2022) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa54385.2022.10032375 · Cited by 1
“Metalearning was first applied to reinforcement learning in 1995 [23], and subsequently various extensions have emerged [24], [25].”
Computational Modeling of Basic Cognitive Functionalities
(2023) · Cambridge University Press eBooks · doi:10.1017/9781108755610.014 · Cited by 1
Uchida, Y., Hikida, T., Honda, M., & Yamashita, Y. (2024) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2024.1502508 · Cited by 1
“First, we conducted the intake-after-food-restriction task by manipulating the inverse temperature parameter β , which is associated with effects of Q -values to the action selections (Schweighofer and Doya, 2003).”
Doya, K. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-15754-7.00088-2 · Cited by 1
Self-citationTwo time scales of adaptation in human learning rates
Simoens, J., Braem, S., Verbeke, P., Chen, H., Mattioni, S., Chai, M., Schuck, N. W., & Verguts, T. (2026) · eLife · doi:10.7554/elife.108223.3 · Cited by 1
Exploiting structure and uncertainty of Bellman updates in Markov decision processes
Tateo, D., D’Eramo, C., Nuara, A., Restelli, M., & Bonarini, A. (2017) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2017.8280923 · Cited by 1
“the dynamical changes of the environment [13], [14], [15].”
BackgroundPreparing to adapt is key for Olympic curling robots
Stork, J. A. (2020) · Science Robotics · doi:10.1126/scirobotics.abe2547 · Cited by 1
Development of Conceptual Framework for Reinforcement Learning Based Optimal Control
Gupta, S., Pal, S., Kumar, K., & Ghosh, K. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-7076-3_29 · Cited by 1
Meta-learning and Neurocomputing – A New Perspective for Computational Intelligence
Castiello, C. (2009) · Studies in computational intelligence · doi:10.1007/978-3-642-01082-8_5 · Cited by 1
“Actually, various meanings can be found in literature coupled with the meta-learning catchword [19, 27, 38, 52].”
BackgroundActive exploration in parameterized reinforcement learningPreprint
Khamassi, M., & Tzafestas, C. S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01986 · Cited by 1
2006 Special Issue Neural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X. (2006) · Cited by 1
Duan, Y. (2017) · eScholarship (California Digital Library) · Cited by 1
Meta-Reinforcement Learning for Heuristic PlanningPreprint
Gutiérrez, R. L., & Leonetti, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02603 · Cited by 1
“…as navigation tasks (Finn, Abbeel, and Levine 2017; Duan et al. 2016; Wang et al. 2017; Mishra et al. 2018), classic control tasks (Li et al. 2018; Schweighofer and Doya 2003; Xu et al. 2018; Sung et al. 2017), and locomotion tasks (Finn, Abbeel, and Levine 2017; Rakelly et al. 2019; Gupta et al.…”
BackgroundKnowledge consolidation and inference in an integrated neuro-cognitive architecture
Oentaryo, R. J. (2011) · doi:10.32657/10356/43793 · Cited by 1
Recasting adaptation as strategy inferencePreprint
Beaumont, S., Khamassi, M., & Domenech, P. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.03.24.645064 · Cited by 2
“Although the behavioral results demonstrate global effects, these can be the consequence of either direct inference of strategies, or global adaptive control of learning parameters in incremental learning models [28, 30].”
Two time scales of adaptation in human learning ratesPreprint
Simoens, J., Braem, S., Verbeke, P., Chen, H., Mattioni, S., Chai, M., Schuck, N. W., & Verguts, T. (2025) · eLife · doi:10.7554/elife.108223.1 · Cited by 1
Meta-learning is expressed through altered prefrontal cortical dynamicsPreprint
Sun, X., Comrie, A. E., Kahn, A. E., Monroe, E. J., Washington, C. B., Joshi, A., Guidera, J. A., Denovellis, E. L., Krausz, T. A., Zhou, J., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.01.697272 · Cited by 1
“This pattern is consistent with knowledge of depletion overriding the initial outcome-driven value updating.”
Learning Nursery Rhymes Using Adaptive Parameter Neurodynamic Programming
Walker, J., & Chalup, S. K. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-14803-8_16 · Cited by 1
Interpreting Dopamine Activities in Stochastic Reward Tasks
Asahina, A., Hirayama, J.-I., & Ishii, S. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02490-0_44
“In this study, we introduce the “metalearning” scheme proposed by Schweighofer and Doya [ 6 ] in the context of RL, to our stochastic reward task (which is not essentially RL).”
BackgroundMethodResultInfluentialComplying with norms. a neurocomputational exploration
Colombo, M. (2012) · ERA
Zarmehri, M. N., & Soares, C. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280656
A Meta-Parameter Learning Method in Reinforcement Learning Based on Temporal Difference Error
Mizoue, H., Kobayashi, K., Kuremoto, T., & Obayashi, M. (2009) · IEEJ Transactions on Electronics Information and Systems · doi:10.1541/ieejeiss.129.1730
Implementation of Cooperative Cognition Under Web Environment
Dou, W., Chen, G., Su, F., & Cai, S. (2005) · Lecture notes in computer science · doi:10.1007/11576259_63
“Question understanding and individual reflection characterizes the metacognition; problem discussion based on knowledge interaction, and cognitive consequence’s refinement and formalization based on meta-cognitions make up of the evolution steps of group cognition process [1, 7, 8].”
BackgroundModel-Free in-situ Optimisation of Data-Ferried Sensor Networks
Pearre, B. W. (2014) · CU Scholar (University of Colorado Boulder) · doi:10.13140/rg.2.1.1965.0169
RESEARCH ARTICLE A Reinforcement Learning Model of Joy, Distress, Hope and Fear
Broekens, J., Jacobs, E., & Jonker, C. M. (2015)
Meta-level reasoning in reinforcement learning
Maissiat, J. (2014) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)
Ottoni, A. L. C., Nepomuceno, E. G., & de Oliveira, M. S. (2017) · ForScience · doi:10.29069/forscience.2017v5n3.e267
Chen, W.-L., Lai, H., Chen, P.-Y., Kan, C., & Lin, C. (2018) · Smart Science · doi:10.1080/23080477.2018.1517293
To Overwrite or to Recall? Individual Differences in Motor AdaptationPreprint
Oh, Y.-M., & Schweighofer, N. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/439406 · Cited by 1
“Kawato, 1998; Haruno et al., 2001; Lee and Schweighofer, 2009; Schweighofer et al., 2011; Lee et al., 2016).”
BackgroundSelf-citationPerformance-Weighed Policy Sampling for Meta-Reinforcement LearningPreprint
Ahmed, I. H., Quiñones-Grueiro, M., & Biswas, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06016
“In this work, we employ meta-RL for faster adaption of the RL policy parameters to collected data samples [19].”
AGILE - a meta learning framework for few-shot brain cell classification
Yuan, P., & Nguyen, H. V. (2022) · Elsevier eBooks · doi:10.1016/b978-0-32-399851-2.00021-1
Computational Cognitive Models of Reinforcement Learning
Doya, K. (2023) · Cambridge University Press eBooks · doi:10.1017/9781108755610.026
Self-citationRegulation of the exploration-exploitation trade-off captures long-term changes in rat behaviourPreprint
Cinotti, F., Coutureau, É., Khamassi, M., Marchand, A. R., & Girard, B. (2023) · doi:10.22541/au.169633848.85039319/v1
da Silva, Í. F., & Ottoni, A. L. C. (2020) · Anais do Simpósio Brasileiro de Pesquisa Operacional · doi:10.59254/sbpo-2020-122695
Methamphetamine-induced adaptation of learning rate dynamics depend on baseline performancePreprint
Kirschner, H., Molla, H. M., Nassar, M. R., de Wit, H., & Ullsperger, M. (2024) · eLife · doi:10.7554/elife.101413.1
Methamphetamine-induced adaptation of learning rate dynamics depend on baseline performancePreprint
Kirschner, H., Molla, H. M., Nassar, M. R., de Wit, H., & Ullsperger, M. (2024) · eLife · doi:10.7554/elife.101413
“Another line of research highlights the role of dopamine in tracking the reward history with multiple learning rates ( Doya, 2002 ; Kolling et al., 2016 ; Meder et al., 2017 ; Schweighofer and Doya, 2003 ).”
Wang, W., Xu, X., Lu, Y., Patnaik, L. M., Li, P., & Cui, L. (2025) · Smart innovation, systems and technologies · doi:10.1007/978-981-97-9124-8_1
Meta-learning for enhancing machines' intelligence in manufacturing
Putnik, G. D. (2025) · Tehnika · doi:10.5937/tehnika2501057p
Robertazzi, F., Vissani, M., & Falotico, E. (2025) · International Journal of Neural Systems · doi:10.1142/s0129065725500406
Methamphetamine-induced adaptation of learning rate dynamics depend on baseline performancePreprint
Kirschner, H., Molla, H. M., Nassar, M. R., de Wit, H., & Ullsperger, M. (2025) · eLife · doi:10.7554/elife.101413.2
Ni, W., Wang, J., Wu, F., Wu, Z., & Wang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5370682
Reinforcement Learning in Robotic Systems: A Review on Sim-to-Real TransferPreprint
Khapre, S. P., Tiwari, R., & Singh, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5387665
Efficient Visual Appearance Optimization by Learning from Prior Preferences
Li, Z., Liao, Y.-C., & Holz, C. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747622
“Meta-learning is a machine learning paradigm that aims to enable models to quickly adapt to new tasks by leveraging prior experience from similar tasks [52, 62].”
Two time scales of adaptation in human learning ratesPreprint
Simoens, J., Braem, S., Verbeke, P., Chen, H., Mattioni, S., Chai, M., Schuck, N. W., & Verguts, T. (2025) · eLife · doi:10.7554/elife.108223
“…the parameters that shape learning is termed meta-learning ( Binz et al., 2024 ; Wang et al., 2018 ), and theories of human meta-learning suggest that people can flexibly adapt their learning rates to the environment ( Mathys et al., 2011 ; Schweighofer and Doya, 2003 ; Silvetti et al., 2018 ).”
Evolving principes of artificial neural design
Wilson, D. G. (2019) · theses.fr (ABES) · doi:10.70675/b678f35dz7ee5z441ez93f3ze801472a9a28
Kouskoff, V. (2021) · theses.fr (ABES) · doi:10.70675/a8f4e479z2341z4f88zaf5bz91e7451d469d
Affective Expert Modulation for Adaptive Decision-Making in Non-stationary Bandits
Kossack, P., & Unger, H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-26319-3_14
Wang, T.-Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7576405
Ni, W., Wang, J., Zeng, J., Wu, Z., & Liang, H. (2026) · Acta Astronautica · doi:10.1016/j.actaastro.2026.10.014
Experimental Study on Parameter Selection for Reinforcement Learning Algorithms
Zajdel, R. (2008) · Theoretical and Applied Informatics
Metadata
- Authors
- Schweighofer, N., & Doya, K.
- Affiliations
- Advanced Telecommunications Research Institute International (JP)
- Corresponding author
- Schweighofer, Nicolas
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2003-01
- Volume
- 16
- Issue
- 1
- Pages
- 5-9
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W1996579288
- Cited by
- 261
- References
- 17
- Citations by year
- 2026: 7 · 2025: 22 · 2024: 20 · 2023: 22 · 2022: 20 · 2021: 26 · 2020: 17 · 2019: 13 · 2018: 16 · 2017: 16 · 2016: 8 · 2015: 7 · 2014: 11 · 2013: 7 · 2012: 6
- FWCI
- 2.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- meta-learning, meta-reinforcement learning, meta-parameters, dopaminergic neurons, parameter adaptation, Markov decision process, dynamic environments
- MeSH terms
- Algorithms, Artificial Intelligence, Decision Theory, Markov Chains, Reinforcement, Psychology, Stochastic Processes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
