The DOI matches its Crossref record; 16 of the 17 references it cites were found in a registry or an index.
Meta-learning in Reinforcement Learning
Abstract
Meta-parameters in reinforcement learning should be tuned to the environmental dynamics and the animal performance. Here, we propose a biologically plausible meta-reinforcement learning algorithm for tuning these meta-parameters in a dynamic, adaptive manner. We tested our algorithm in both a simulation of a Markov decision task and in a non-linear control task. Our results show that the algorithm robustly finds appropriate meta-parameter values, and controls the meta-parameter time course, in both static and dynamic environments. We suggest that the phasic and tonic components of dopamine neuron firing can encode the signal required for meta-learning of reinforcement learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564
Aston-Jones, G., & Bloom, F. (1981) · The Journal of Neuroscience · doi:10.1523/jneurosci.01-08-00876.1981 · Cited by 1,960
An opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 995
Self-citationOpponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844
Metalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704
Self-citation5-HT and motor control: a hypothesis
Jacobs, B. L., & Fornal, C. A. (1993) · Trends in Neurosciences · doi:10.1016/0166-2236(93)90090-9 · Cited by 452
On the Complexity of Solving Markov Decision ProblemsPreprint
Littman, M. L., Dean, T., & Kaelbling, L. P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4971 · Cited by 434
A stochastic reinforcement learning algorithm for learning real-valued functions
Gullapalli, V. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90056-q · Cited by 292
The role of exploration in learning control
Thrun, S. (1992) · Cited by 226
Research works printed but not linked: 1
- Tanaka, 2002, Functional MRI study of short-term and long-term prediction of reward, Proceedings of the Eighth International Conference on Functional Mapping of the Human Brain, Sendai, Japan, 1062
Cited by
Sorted by citations
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
Deep Reinforcement Learning: A Survey
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., & Miao, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3207346 · Cited by 972
“The main idea of meta-reinforcement learning [149] is to apply the prior knowledge learned from a large number of RL tasks to new RL tasks, and to improve the learning speed and generalization ability of the agent.”
BackgroundReinforcement learning, fast and slow
Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., & Hassabis, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.02.006 · Cited by 667
“Meta-learning occurs when one learning system progressively adjusts the operation of a second learning system, such that the latter operates with increasing speed and efficiency [83, 84, 85, 86, 87].”
Neuroeconomics: Decision Making and the Brain
Rushworth, M. F. S., & Walton, M. E. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.005 · Cited by 669
Learning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579
“…for meeting these challenges, which we refer to as deep meta-reinforcement learning , a label that is intended to both link it with and distinguish it from previous work employing the term “meta-reinforcement learning” (e.g. Schmidhuber et al., 1996; Schweighofer and Doya, 2003, discussed later).”
BackgroundPrefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596
One-shot Learning with Memory-Augmented Neural NetworksPreprint
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06065 · Cited by 515
“Although the term has been used in numerous senses (Schmidhuber et al., 1997; Caruana, 1997; Schweighofer & Doya, 2003; Brazdil et al., 2003), meta-learning generally refers to a scenario in which an agent learns at two levels, each associated with different time scales.”
MethodMedial and orbital frontal cortex in decision-making and flexible behavior
Klein-Flügge, M. C., Bongioanni, A., & Rushworth, M. F. S. (2022) · Neuron · doi:10.1016/j.neuron.2022.05.022 · Cited by 244
Deep reinforcement learning in computer vision: a comprehensive survey
Le, N., Rathour, V. S., Yamazaki, K., Luu, K., & Savvides, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10061-9 · Cited by 271
Low-Serotonin Levels Increase Delayed Reward Discounting in Humans
Schweighofer, N., Bertin, M., Shishida, K., Okamoto, Y., Tanaka, S., Yamawaki, S., & Doya, K. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.4982-07.2008 · Cited by 265
“Therefore, to maximize total gain in such tasks, the discount rate should be carefully adjusted (Schweighofer and Doya, 2003; Schweighofer et al., 2006).”
BackgroundSelf-citationThe Exploration-Exploitation Dilemma: A Multidisciplinary Framework
Berger‐Tal, O., Nathan, J., Meroni, E., & Saltz, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0095693 · Cited by 256
“Moreover, analogical neurophysiologic pathways in the brain of animals and humans have been suggested, highlighting the neurobiological substrates that are related to the regulation of decision-making [17,18,20].”
BackgroundDeep reinforcement learning in medical imaging: A literature review
Zhou, S., Le, N., Luu, K., Nguyen, H. V., & Ayache, N. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102193 · Cited by 227
Dual Adaptation Supports a Parallel Architecture of Motor Memory
Lee, J.-Y., & Schweighofer, N. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1294-09.2009 · Cited by 202
“, metalearning (Schweighofer and Doya, 2003)].”
MethodSelf-citationFusi, S., Asaad, W. F., Miller, E. K., & Wang, X.-J. (2007) · Neuron · doi:10.1016/j.neuron.2007.03.017 · Cited by 199
“Memory performances are higher with modifiable learning rates than with a static distribution of learning rates because each synapse changes adaptively the rules by which it is modified (metalearning) (Schweighofer and Doya, 2003; Soltani et al., 2006).”
BackgroundSalvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200
“A promising research direction to tackle the RG problem could be the meta-learning strategy application in RL, in order to quickly adapt experience gained in simulation on the real system [115], [116].”
BackgroundTwenty-Five Lessons from Computational Neuromodulation
Dayan, P. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.027 · Cited by 190
Meta-learning in natural and artificial intelligence
Wang, J. X. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.01.002 · Cited by 171
Emotion in reinforcement learning agents and robots: a survey
Moerland, T. M., Broekens, J., & Jonker, C. M. (2018) · Machine Learning · doi:10.1007/s10994-017-5666-0 · Cited by 205
“(2007) Gridworld agent Maze Learning Emotional agent cannot improve results of (Broekens et al, 2007b; Schweighofer and Doya, 2003) Shi et al.”
BackgroundInfluentialTeaching recurrent neural networks to infer global temporal structure from local examples
Kim, J. Z., Lu, Z., Nozari, E., Pappas, G. J., & Bassett, D. S. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00321-2 · Cited by 146
Reinforcement learning for HVAC control in intelligent buildings: A technical and conceptual review
Al Sayed, K., Boodi, A., Sadeghian Broujeny, R., & Beddiar, K. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.110085 · Cited by 140
A Biophysically Based Neural Model of Matching Law Behavior: Melioration by Stochastic Synapses
Soltani, A., & Wang, X. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.5159-05.2006 · Cited by 129
“This raises the interesting question of whether learning rates themselves should be plastic (meta-learning) and adjustable according to behavioral demands (Doya, 2002; Schweighofer and Doya, 2003).”
BackgroundDorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner
Silvetti, M., Vassena, E., Abrahamse, E. L., & Verguts, T. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006370 · Cited by 140
Qiao, H., Chen, J., & Huang, X. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3071312 · Cited by 121
“A part of works suggests that positive and negative emotions can be coded by the value function directly [99], [103], or be computed from the temporal change of reward sequence [104].”
BackgroundFederated Learning and Meta Learning: Approaches, Applications, and Directions
Liu, X., Deng, Y., Nallanathan, A., & Bennis, M. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3330910 · Cited by 116
“…devices having different data distributions but performing the same task, meta learning guarantees the adaptation of models to multiple tasks with their corresponding datasets, often emphasizing fast adaptation and generalization capabilities, especially by initialization-based algorithms [40].”
BackgroundMetaplasticity as a Neural Substrate for Adaptive Learning and Choice under Uncertainty
Farashahi, S. G., Donahue, C. H., Khorsand, P., Seo, H., Lee, D., & Soltani, A. (2017) · Neuron · doi:10.1016/j.neuron.2017.03.044 · Cited by 106
“reward feedback: (1) to determine the optimal learning rates based on optimization or on the level of uncertainty in a given environment (RL models) (Doya, 2002; Schweighofer and Doya, 2003; Preuschoff and Bossaerts, 2007); (2) to identify the correct model of the world, which includes the amount of uncertainty to pro…”
Adaptive properties of differential learning rates for positive and negative outcomes
Cazé, R. D., & van der Meer, M. A. A. (2013) · Biological Cybernetics · doi:10.1007/s00422-013-0571-5 · Cited by 103
“and compatible with (within a single RL system) meta-learning of other RL parameters [10,33].”
BackgroundDopamine blockade impairs the exploration-exploitation trade-off in rats
Cinotti, F., Fresno, V., Aklil, N., Coutureau, É., Girard, B., Marchand, A. R., & Khamassi, M. (2019) · Scientific Reports · doi:10.1038/s41598-019-43245-z · Cited by 105
Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning Model
Khamassi, M., Lallée, S., Enel, P., Procyk, E., & Dominey, P. F. (2011) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2011.00001 · Cited by 104
Sex differences in learning from exploration
Chen, C. S., Knep, E., Han, A., Ebitz, R. B., & Grissom, N. M. (2021) · eLife · doi:10.7554/elife.69748 · Cited by 105
Khamassi, M., Quilodran, R., Enel, P., Dominey, P. F., & Procyk, E. (2014) · Cerebral Cortex · doi:10.1093/cercor/bhu114 · Cited by 95
“, shifting between exploration and exploitation) is achieved by dynamically tuning meta-parameters based on measures of the agent’s performance (Doya 2002; Ishii et al. 2002; Schweighofer and Doya 2003).”
MethodUse It and Improve It or Lose It: Interactions between Arm Function and Use in Humans Post-stroke
Hidaka, Y., Han, C. E., Wolf, S. L., Winstein, C. J., & Schweighofer, N. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002343 · Cited by 88
“, [35,36]).”
Self-citationKusy, M., & Zajdel, R. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2376703 · Cited by 86
“Its large value introduces the oscillations of the Q elements [38].”
BackgroundEfficient reinforcement learning: computational theories, neuroscience and robotics
Kawato, M., & Samejima, K. (2007) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2007.03.004 · Cited by 82
The new robotics—towards human‐centered machines
Schaal, S. (2007) · HFSP journal · doi:10.2976/1.2748612 · Cited by 87
“According to a preliminary model (Daw and Touretzky, 2002; Schweighofer and Doya, 2003; Schweig-hofer et al. , 2004), the motivation to learn will depend on the balance between successes, failures, and external rewards.”
BackgroundMethodHumans Can Adopt Optimal Discounting Strategy under Real-Time Constraints
Schweighofer, N., Shishida, K., Han, C. E., Okamoto, Y., Tanaka, S., Yamawaki, S., & Doya, K. (2006) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0020152 · Cited by 87
“Thus, the discount rate should be carefully adjusted to maximize total gain in task situations of repeated forced choices with varying delays to the rewards and limited total time [14,15].”
BackgroundSelf-citationGautam, M. (2023) · Electricity · doi:10.3390/electricity4040020 · Cited by 94
“Furthermore, meta-learning, as discussed in [130], involves optimizing meta-parameters in RL algorithms to facilitate more efficient learning.”
MethodNeural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X.-J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.044 · Cited by 77
“In reinforcement learning, meta-learning algorithms are commonly evoked to adjust parameters, such as the learning rate for updating value functions and the rate of temporal discounting for delayed rewards (Doya, 2002; Schweighofer & Doya, 2003).”
BackgroundMethodMeta-seg: A survey of meta-learning for image segmentation
Luo, S., Li, Y., Gao, P., Wang, Y., & Serikawa, S. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108586 · Cited by 79
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73
“If you are interested, further exploration can be found in upcoming surveys [115], [116], [117], [118], [119], [120], [121].”
Najar, A., Bonnet, E., Bahrami, B., & Palminteri, S. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3001028 · Cited by 75
“At the computational level, this effect can be formalized as a meta-learning process, where relevant variables are used to optimally tune the learning parameters [29,30].”
BackgroundMeta-control: From psychology to computational neuroscience
Eppinger, B., Goschke, T., & Musslick, S. (2021) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-021-00919-4 · Cited by 77
Meta-reinforcement learning via orbitofrontal cortex
Hattori, R., Hedrick, N. G., Jain, A., Chen, S., You, H., Hattori, M., Choi, J., Lim, B. K., Yasuda, R., & Komiyama, T. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01485-3 · Cited by 75
“Furthermore, plasticity-based RL may happen at different speeds in different brain areas, possibly governed by differences of neuromodulatory inputs and their receptor expression 5,23,25,50 .”
BackgroundMethodCortical mechanisms for reinforcement learning in competitive games
Seo, H., & Lee, D. (2008) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2008.0158 · Cited by 66
Berniker, M., & Kording, K. P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002210 · Cited by 66
Serotonin and the Evaluation of Future Rewards
Schweighofer, N., Tanaka, S., & Doya, K. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.011 · Cited by 65
Self-citationReinforcement learning for the traveling salesman problem with refueling
Ottoni, A. L. C., Nepomuceno, E. G., de Oliveira, M. S., & de Oliveira, D. C. R. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00444-4 · Cited by 60
Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters
Khamassi, M., Enel, P., Dominey, P. F., & Procyk, E. (2013) · Progress in brain research · doi:10.1016/b978-0-444-62604-2.00022-8 · Cited by 67
An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning
Potjans, W., Diesmann, M., & Morrison, A. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001133 · Cited by 56
“Self-adjusting parameters have also been implemented in other three-factor learning rules such as the one in [53] based on the meta-learning algorithm proposed in [54].”
DAPath: Distance-aware knowledge graph reasoning based on deep reinforcement learning
Tiwari, P., Zhu, H., & Pandey, H. M. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.11.012 · Cited by 55
“[1] propose a reinforcement learning (RL) [2, 3, 4, 5, 6, 7, 8] method, DeepPath.”
MethodGASTO: A Fast Adaptive Graph Learning Framework for Edge Computing Empowered Task Offloading
Li, Y., Li, J., Lv, Z., Li, H., Wang, Y., & Xu, Z. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3250395 · Cited by 56
“However, meta-reinforcement learning [11], [12] can learn more efficient learning algorithms than learning from scratch.”
BackgroundBroekens, J., Kosters, W. A., & Verbeek, F. J. (2007) · Adaptive Behavior · doi:10.1177/1059712307084686 · Cited by 49
“Therefore, the agent’s performance is defined by the difference between the long-term average reinforcement signal (“what am I used to”) and the short-term average reinforcement signal (“how am I doing now”; cf. Schweighofer & Doya, 2003).”
BackgroundBrain-Inspired Learning, Perception, and Cognition: A Comprehensive Review
Jiao, L., Ma, M., He, P., Geng, X., Liu, X., Liu, F., Ma, W., Yang, S., Hou, B., & Tang, X. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3401711 · Cited by 54
“Meta-learning was first utilized in the context of RL in 1995 [129].”
Catecholaminergic modulation of meta-learning
Cook, J. L., Swart, J. C., Froböse, M. I., Diaconescu, A. O., Geurts, D. E. M., den Ouden, H. E. M., & Cools, R. (2019) · eLife · doi:10.7554/elife.51439 · Cited by 54
A reinforcement learning model of joy, distress, hope and fear
Broekens, J., Jacobs, E., & Jonker, C. M. (2015) · Connection Science · doi:10.1080/09540091.2015.1031081 · Cited by 48
“Studying the relation be-tween RL and affect is no exception (see e.g. (Schweighofer & Doya, 2003; Cos, Canamero, Hayes, & Gillies, 2013; Krichmar, 2008; Redish, Jensen, Johnson, & Kurth-Nelson, 2007)).”
BackgroundInfluentialKhamassi, M., Velentzas, G., Tsitsimis, T., & Tzafestas, C. S. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2843122 · Cited by 43
“Thus here, following the proposition of [26], we measure a long-term reward running average ̄̄ rt serving as reference, and a short-term one r̄t serving as current measure of performance.”
BackgroundMethodInfluentialKusy, M., & Zajdel, R. (2014) · Applied Intelligence · doi:10.1007/s10489-014-0562-9 · Cited by 41
“Its large value introduces the oscillations of Q elements [34].”
BackgroundFederici, L., Scorsoglio, A., Zavoli, A., & Furfaro, R. (2022) · Acta Astronautica · doi:10.1016/j.actaastro.2022.08.047 · Cited by 44
Adaptable automation with modular deep reinforcement learning and policy transfer
Raziei, Z., & Moghaddam, M. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104296 · Cited by 40
Goals, usefulness and abstraction in value-based choice
Martino, B. D., & Cortese, A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.11.001 · Cited by 46
“The cortico-thalamic loops linking the PFC and basal ganglia/thalamusprovidethe neuralunderpinningformeta-learning [133,134].”
BackgroundMeta-learning within Projective Simulation
Makmal, A., Melnikov, A. A., Dunjko, V., & Briegel, H. J. (2016) · IEEE Access · doi:10.1109/access.2016.2556579 · Cited by 36
“An example for such a reflex-type adaptation approach for meta-learning can be found in [15] where the fundamental RL parameters, namely, the learning rate α, the exploitationexploration parameter β, and the discount factor γ are tuned according to predefined equations; In contrast, an agent”
MethodExplainable Artificial Intelligence (xAI) Approaches and Deep Meta-Learning Models
Dağlarlı, E. (2020) · IntechOpen eBooks · doi:10.5772/intechopen.92172 · Cited by 38
Takemoto, A., Miwa, M., Koba, R., Yamaguchi, C., Suzuki, H., & Nakamura, K. (2014) · Neuroscience Research · doi:10.1016/j.neures.2014.10.001 · Cited by 36
Barsce, J. C., Palombarini, J. A., & Martínez, E. (2017) · CLEI electronic journal · doi:10.1109/clei.2017.8226439 · Cited by 37
“A metalearning approach has been proposed in [23] to tune the hyper-parameters of a reinforcement learning for a problem related to animal behavior.”
BackgroundTuning of reinforcement learning parameters applied to SOP using the Scott–Knott method
Ottoni, A. L. C., Nepomuceno, E. G., de Oliveira, M. S., & de Oliveira, D. C. R. (2019) · Soft Computing · doi:10.1007/s00500-019-04206-w · Cited by 34
Multi-Agent Meta-Reinforcement Learning for Self-Powered and Sustainable Edge Computing Systems
Munir, M. S., Tran, N. H. K., Saad, W., & Hong, C. S. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3057960 · Cited by 37
A Comprehensive Overview and Survey of Recent Advances in Meta-LearningPreprint
Peng, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11149 · Cited by 58
“In [16], meta-learning is integrated in RL to tune hyper-parameters such as learning rate, exploration-exploitation tradeoff and discount factor of future reward.”
Learning environment-specific learning rates
Simoens, J., Verguts, T., & Braem, S. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011978 · Cited by 38
Li, X., Zhu, R., Ye, H., Jiang, C., & Benslimane, A. (2021) · Safety Science · doi:10.1016/j.ssci.2021.105315 · Cited by 31
Khamassi, M., Velentzas, G., Tsitsimis, T., & Tzafestas, C. S. (2017) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc.2017.33 · Cited by 29
“Thus here, following the proposition of [9], we measure a long-term reward running average r̄t serving as reference, and a short-term one r̄t serving as current measure of performance.”
BackgroundMethodInfluentialImage-based Deep Reinforcement Learning for Autonomous Lunar Landing
Scorsoglio, A., Furfaro, R., Linares, R., & Gaudet, B. (2020) · AIAA Scitech 2020 Forum · doi:10.2514/6.2020-1910 · Cited by 31
Gao, M., Feng, X., Yu, H., & Li, X. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04574-9 · Cited by 28
“the advantage function shows the improvement in the expected cumulative reward gained when taking action in the state St , compared to the average of all possible actions taken in the state St [39].”
Sun, Y., Pang, S., Zhao, Z., & Zhang, Y.-A. (2024) · Natural Resources Research · doi:10.1007/s11053-024-10396-4 · Cited by 28
Computational Modeling of Catecholamines Dysfunction in Alzheimer’s Disease at Pre-Plaque Stage
Caligiore, D., Silvetti, M., D’Amelio, M., Puglisi‐Allegra, S., & Baldassarre, G. (2020) · Journal of Alzheimer s Disease · doi:10.3233/jad-200276 · Cited by 30
“Learning to optimally control these internal variables is defined as meta-learning [36, 37].”
BackgroundMetaFollower: Adaptable personalized autonomous car following
Chen, X., Chen, K., Zhu, M., Yang, H., Shen, S., Wang, X., & Wang, Y. (2024) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2024.104872 · Cited by 31
Relative salience signaling within a thalamo-orbitofrontal circuit governs learning rate
Namboodiri, V. M. K., Hobbs, T. G., Trujillo‐Pisanty, I., Simon, R. C., Hjort, M. M., & Stuber, G. D. (2021) · Current Biology · doi:10.1016/j.cub.2021.09.037 · Cited by 26
Sugiyama, T., Schweighofer, N., & Izawa, J. (2023) · Nature Communications · doi:10.1038/s41467-023-39536-9 · Cited by 30
“Although such a hierarchical learning system is thought to exist in the brain 44 – 48 , direct empirical evidence of its existence is lacking.”
Self-citationStrategies for Affect-Controlled Action-Selection in Soar-RL
Hogewoning, E., Broekens, J., Eggermont, J., & Bovenkamp, E. G. P. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73055-2_52 · Cited by 26
“On the switch task, SD performed worse than expected, given that successful use was reported on another task [10].”
BackgroundMethodInfluentialIntrinsic Motivation and Introspection in Reinforcement Learning
Merrick, K. E. (2012) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2012.2208457 · Cited by 26
“Approaches to cognitive selectivity in RL include meta-learning models for automatically tuning learning parameters in RL [12].”
MethodBehavioral Game Theory and the Neural Basis of Strategic Choice
Camerer, C. F. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00013-0 · Cited by 27
On Affect and Self-adaptation: Potential Benefits of Valence-Controlled Action-Selection
Broekens, J., Kosters, W. A., & Verbeek, F. J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73053-8_36 · Cited by 26
“Schweighofer and Doya [17] used a similar measure for ”how well the agent is doing compared to what it is used to”; however, they use it differently.”
BackgroundFast Task Adaptation Based on the Combination of Model-Based and Gradient-Based Meta Learning
Xu, Z., Chen, X., & Cao, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3028378 · Cited by 26
“Recent success in the development of metalearning [3] provides DRL methods an efficient way to quickly adapt to unseen tasks with only a few interactions through learning from a distribution of related yet different tasks.”
MethodAnalysis of Amino Acid Mixtures by Voltammetric Electronic Tongues and Artificial Neural Networks
Faura, G., González‐Calabuig, A., & del Valle, M. (2016) · Electroanalysis · doi:10.1002/elan.201600055 · Cited by 25
Best to go with what you know?
Lee, D. (2006) · Nature · doi:10.1038/441822a · Cited by 25
C2S : Class-aware client selection for effective aggregation in federated learning
Cao, M. Q., Zhang, Y., Ma, Z., & Zhao, M. (2022) · High-Confidence Computing · doi:10.1016/j.hcc.2022.100068 · Cited by 24
Dağlarlı, E. (2020) · Advances in systems analysis, software engineering, and high performance computing book series · doi:10.4018/978-1-7998-5101-1.ch003 · Cited by 23
Online meta-learning by parallel algorithm competition
Elfwing, S., Uchibe, E., & Doya, K. (2018) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3205455.3205486 · Cited by 23
“ry approach (i.e., the learning started from scratch in each generation) to find appropriate fixed values of the meta-parameters (Unemi et al., 1994; Eriksson et al., 2003; Elfwing et al., 2008, 2011). Schweighofer and Doya (2003) proposed a meta-learning method based on the mid-term and the long-term running averages o…”
MethodSelf-citationRobust interplanetary trajectory design under multiple uncertainties via meta-reinforcement learning
Federici, L., & Zavoli, A. (2023) · Acta Astronautica · doi:10.1016/j.actaastro.2023.10.018 · Cited by 23
Ottoni, A. L. C., Nepomuceno, E. G., & de Oliveira, M. S. (2018) · Journal of Control Automation and Electrical Systems · doi:10.1007/s40313-018-0374-y · Cited by 20
“Murakoshi and Mizuno (2004), based on the work of Schweighofer and Doya (2003), suggest a parameter control method that responds more quickly to sudden changes in the environment.”
BackgroundFereoli, G., Schaub, H., & Lizia, P. D. (2024) · Journal of Spacecraft and Rockets · doi:10.2514/1.a36100 · Cited by 23
Robertazzi, F., Vissani, M., Schillaci, G., & Falotico, E. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.06.020 · Cited by 19
An integrative control theory perspective on consciousness.
Mansell, W. (2022) · Psychological Review · doi:10.1037/rev0000384 · Cited by 21
Reinforcement learning with state-dependent discount factor
Yoshida, N., Uchibe, E., & Doya, K. (2013) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2013.6652533 · Cited by 20
Self-citationFernández, F. C., & Caarls, W. (2018) · International Conference on Information Systems and Computer Science (INCISCOS) · doi:10.1109/inciscos.2018.00050 · Cited by 19
“[10] presents a different approach using biological-based neuromodulators to tune hyperparameters in an adaptive form”
MethodReinforcement learning approaches to hippocampus-dependent flexible spatial navigation
Tessereau, C., O’Dea, R. D., Coombes, S., & Bast, T. (2021) · Brain and Neuroscience Advances · doi:10.1177/2398212820975634 · Cited by 19
Boosting Few-Shot Image Recognition Via Domain Alignment Prototypical Networks
Lu, J., Cao, Z. L., Wu, K., Zhang, G., & Zhang, C. (2018) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2018.00048 · Cited by 16
“Some works advised to tackle this problem from the perspective of meta-learning [16]–[18], which involves a meta-learner that tries to quickly learn a classifier on novel classes given a few training data for this new task [19]–[21].”
BackgroundKan, C., Chen, W., Lin, C., Wang, J., Lu, P.-J., Chan, M.-Y., & Wu, J.-T. (2018) · IEEE Access · doi:10.1109/access.2018.2815599 · Cited by 17
Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks
Zhang, L., Zhou, Y., Yang, Y., & Gao, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3385745 · Cited by 18
Online Learning of a Memory for Learning Rates
Meier, F., Kappler, D., & Stefan Schaal, A. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460625 · Cited by 17
Learning continuous chaotic attractors with a reservoir computer
Smith, L. M., Kim, J. Z., Lu, Z., & Bassett, D. S. (2022) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0075572 · Cited by 18
Metadata
- Authors
- Schweighofer, N., & Doya, K.
- Affiliations
- Advanced Telecommunications Research Institute International (JP)
- Corresponding author
- Schweighofer, Nicolas
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2003-01
- Volume
- 16
- Issue
- 1
- Pages
- 5-9
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W1996579288
- Cited by
- 261
- References
- 17
- Citations by year
- 2026: 7 · 2025: 22 · 2024: 20 · 2023: 22 · 2022: 20 · 2021: 26 · 2020: 17 · 2019: 13 · 2018: 16 · 2017: 16 · 2016: 8 · 2015: 7 · 2014: 11 · 2013: 7 · 2012: 6
- FWCI
- 2.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Core Research for Evolutional Science and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- meta-learning, meta-reinforcement learning, meta-parameters, dopaminergic neurons, parameter adaptation, Markov decision process, dynamic environments
- MeSH terms
- Algorithms, Artificial Intelligence, Decision Theory, Markov Chains, Reinforcement, Psychology, Stochastic Processes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
