Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning
Open accessAbstract
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some recent works on quickly adapting to changes in the data stream had offline problem settings with two separate phases, one for learning and the other for evaluation.
“However, the problem settings in these papers were offline as they had two separate phases, one for learning and the other for evaluation.”
From Loss of plasticity in deep continual learning (2024) · cites this paper as [122]
Section 4 (p. 5) distinguishes a train-time algorithm from a test-time algorithm, supporting the claim that there are separate phases. But the test-time algorithm adapts the model at every timestep, and Section 6.4 (p. 9) explicitly describes this as online adaptation. Calling the paper's problem setting offline is therefore not supported. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · see citing works
Åström, K. J., & Wittenmark, B. (1989) · Cited by 5,009 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
Self-citationHuman-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · see citing works
Optimization as a Model for Few-Shot Learning
Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Self-citationLearning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & de Freitas, N. (2016) · arXiv · doi:10.48550/arxiv.1606.04474 · Cited by 1,486 · see citing works
Adaptive control of linearizable systems
Sastry, S., & Isidori, A. (1989) · IEEE Transactions on Automatic Control · doi:10.1109/9.40741 · Cited by 1,195 · see citing works
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136 · see citing works
A Simple Neural Attentive Meta-LearnerPreprint
Mishra, N., Rohaninejad, M., Chen, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03141 · Cited by 961 · see citing works
Self-citationNeural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8463189 · Cited by 898 · see citing works
Self-citationReinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831 · see citing works
Continuous Deep Q-Learning with Model-based AccelerationPreprint
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 770 · see citing works
Self-citationLearning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · see citing works
Learning to Learn: Introduction and Overview
Thrun, S., & Pratt, L. (1998) · Learning to Learn · doi:10.1007/978-1-4615-5529-2_1 · Cited by 521 · see citing works
Online Parameter Estimation and Adaptive Control of Permanent-Magnet Synchronous Machines
Underwood, S., & Husain, I. (2009) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2009.2036029 · Cited by 502 · see citing works
Information theoretic MPC for model-based reinforcement learning
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., & Theodorou, E. A. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989202 · Cited by 495 · see citing works
Levine, S., & Koltun, V. (2013) · Cited by 481 · see citing works
Self-citationOne-shot Learning with Memory-Augmented Neural NetworksPreprint
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06065 · Cited by 435 · see citing works
Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.131 · Cited by 357 · see citing works
GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models
Ko, J., & Fox, D. (2009) · Autonomous Robots · doi:10.1007/s10514-009-9119-x · Cited by 342 · see citing works
DeepMPC: Learning Deep Latent Features for Model Predictive Control
Lenz, I., Knepper, R. A., & Saxena, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.012 · Cited by 339 · see citing works
Model-Ensemble Trust-Region Policy OptimizationPreprint
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10592 · Cited by 276 · see citing works
Self-citationContinuous Adaptation via Meta-Learning in Nonstationary and Competitive EnvironmentsPreprint
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03641 · Cited by 263 · see citing works
Self-citationMultiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · see citing works
Online movement adaptation based on previous sensor experiences
Pástor, P., Righetti, L., Kalakrishnan, M., & Stefan Schaal, A. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6095059 · Cited by 224 · see citing works
Meta-neural networks that learn by learning
Naik, D., & Mammone, R. J. (2003) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.287172 · Cited by 191 · see citing works
Finn, C., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11622 · Cited by 160 · see citing works
Self-citationLearning to OptimizePreprint
Li, K., & Malik, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01885 · Cited by 159 · see citing works
Bayesian Recurrent Neural Networks
Fortunato, M., Blundell, C., & Vinyals, O. (2017) · arXiv · doi:10.48550/arxiv.1704.02798 · Cited by 145 · see citing works
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors
Fu, J., Levine, S., & Abbeel, P. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759592 · Cited by 126 · see citing works
Self-citationLearning Optimal Adaptation Strategies in Unpredictable Motor Tasks
Braun, D. A., Aertsen, A., Wolpert, D. M., & Mehring, C. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.3075-08.2009 · Cited by 110 · see citing works
Meta Reinforcement Learning with Latent Variable Gaussian Processes
Sæmundsson, S., Hofmann, K., & Deisenroth, M. P. (2018) · arXiv · doi:10.48550/arxiv.1803.07551 · Cited by 106 · see citing works
Learning to Learn: Meta-Critic Networks for Sample Efficient LearningPreprint
Sung, F., Li, Z., Xiang, T., Hospedales, T. M., & Yang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09529 · Cited by 91 · see citing works
Dynamic Evaluation of Neural Sequence Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2017) · arXiv · doi:10.48550/arxiv.1709.07432 · Cited by 90 · see citing works
Optimization of Perturbative PV MPPT Methods Through Online System Identification
Manganiello, P., Ricco, M., Petrone, G., Monmasson, É., & Spagnuolo, G. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2014.2317143 · Cited by 77 · see citing works
Model Predictive Path Integral Control using Covariance Variable Importance Sampling
Williams, G., Aldrich, A., & Theodorou, E. (2015) · arXiv · doi:10.48550/arxiv.1509.01149 · Cited by 71 · see citing works
LEARNING TO GENERATE ARTIFICIAL FOVEA TRAJECTORIES FOR TARGET DETECTION
Schmidhuber, J., & Huber, R. (1991) · International Journal of Neural Systems · doi:10.1142/s012906579100011x · Cited by 68 · see citing works
Aswani, A. J., Bouffard, P., & Tomlin, C. J. (2012) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2012.6315483 · Cited by 68 · see citing works
Meta-learning with backpropagation
Younger, A. S., Hochreiter, S., & Conwell, P. R. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2001.938471 · Cited by 64 · see citing works
Online System Identification and Adaptive Control for PEM Fuel Cell Maximum Efficiency Tracking
Kélouwani, S., Adegnon, K. M., Agbossou, K., & Dubé, Y. (2012) · IEEE Transactions on Energy Conversion · doi:10.1109/tec.2012.2194496 · Cited by 56 · see citing works
Learning feedback terms for reactive planning and control
Rai, A., Sutanto, G., Stefan Schaal, A., & Meier, F. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989252 · Cited by 50 · see citing works
Towards robust online inverse dynamics learning
Meier, F., Kappler, D., Ratliff, N., & Stefan Schaal, A. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759594 · Cited by 41 · see citing works
Adaptive model predictive control for constrained linear systems
Tanasković, M., Fagiano, L., Smith, R. S., Goulart, P. J., & Morari, M. (2013) · European Control Conference (ECC) · doi:10.23919/ecc.2013.6669544 · Cited by 38 · see citing works
Model-based policy search for automatic tuning of multivariate PID controllers
Doerr, A., Nguyen-Tuong, D., Marco, A., Stefan Schaal, A., & Trimpe, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989622 · Cited by 33 · see citing works
Learning Image-Conditioned Dynamics Models for Control of Underactuated Legged Millirobots
Nagabandi, A., Yang, G., Asmar, T., Pandya, R., Kahn, G., Levine, S., & Fearing, R. S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594193 · Cited by 32 · see citing works
Self-citationDrifting Gaussian processes with varying neighborhood sizes for online model learning
Meier, F., & Stefan Schaal, A. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487143 · Cited by 28 · see citing works
Learning Rapid-Temporal AdaptationsPreprint
Munkhdalai, T., Yuan, X., Mehri, S., Wang, T., & Trischler, A. (2017) · arXiv (Cornell University) · Cited by 10 · see citing works
Structure Learning in Motor Control:A Deep Reinforcement Learning ModelPreprint
Weinstein, A., & Botvinick, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06827 · Cited by 8 · see citing works
Online representation learning in recurrent neural language models
Rei, M. (2015) · doi:10.17863/cam.21361 · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 , 2016. · arXiv
- K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114 , 2018. · arXiv
- M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foundations and Trends R © in Robotics , 2(1-2):1-142, 2013.
- Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning. CoRR , abs/1611.02779, 2016. · arXiv
- C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR , abs/1703.03400, 2017. · arXiv
- T. Munkhdalai and H. Yu. Meta networks. arXiv preprint arXiv:1703.00837 , 2017. · arXiv
Cited by
Sorted by citations
Deep Reinforcement Learning: A Survey
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., & Miao, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3207346 · Cited by 972 · see citing works
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886 · see citing works
Learning-Based Model Predictive Control: Toward Safe Learning in Control
Hewing, L., Wabersich, K. P., Menner, M., & Zeilinger, M. N. (2019) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-090419-075625 · Cited by 849 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Self-citationModel-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 668 · see citing works
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively, H., Bellathur, A., Hausman, K., Finn, C., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1910.10897 · Cited by 450 · see citing works
Self-citationA practical guide to multi-objective reinforcement learning and planning
Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., et al. (2022) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-022-09552-y · Cited by 400 · see citing works
Learning Agile Robotic Locomotion Skills by Imitating Animals
Bin Peng, X., Coumans, E., Zhang, T., Lee, T.-W., Tan, J., & Levine, S. (2020) · Robotics: Science and Systems XVI · doi:10.15607/rss.2020.xvi.064 · Cited by 357 · see citing works
Self-citationA survey of deep meta-learning
Huisman, M., van Rijn, J. N., & Plaat, A. (2021) · Data Archiving and Networked Services (DANS) · doi:10.1007/s10462-021-10004-4#rightslink · Cited by 316 · see citing works
Neural-Fly enables rapid learning for agile flight in strong winds
O’Connell, M., Shi, G., Shi, X., Azizzadenesheli, K., Anandkumar, A., Yue, Y., & Chung, S. (2022) · Science Robotics · doi:10.1126/scirobotics.abm6597 · Cited by 297 · see citing works
Deep reinforcement learning: a survey
Wang, H.-N., Liu, N., Zhang, Y., Feng, D., Huang, F., Li, D., & Zhang, Y. (2020) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1900533 · Cited by 286 · see citing works
Deep reinforcement learning in computer vision: a comprehensive survey
Le, N., Rathour, V. S., Yamazaki, K., Luu, K., & Savvides, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10061-9 · Cited by 265 · see citing works
Abbasloo, S., Yen, C.-Y., & Chao, H. J. (2020) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3387514.3405892 · Cited by 253 · see citing works
Deep reinforcement learning in medical imaging: A literature review
Zhou, S., Le, N., Luu, K., Nguyen, H. V., & Ayache, N. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102193 · Cited by 222 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
“(Nagabandi et al., 2019a, GrBAL,ReBAL) use meta-RL methods (MAML Finn et al. (2017) and RL2 Duan et al. (2016)) to learn to quickly adapt a dynamics model using gradients or recurrence, and then use the adapted model to plan.”
MethodHu, S., Chen, X., Ni, W., Hossain, E., & Wang, X. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3086014 · Cited by 178 · see citing works
MetaLight: Value-Based Meta-Reinforcement Learning for Traffic Signal Control
Zang, X., Yao, H., Zheng, G., Xu, N., Xu, K. M., & Li, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5467 · Cited by 172 · see citing works
Loss of plasticity in deep continual learning
Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024) · Nature · doi:10.1038/s41586-024-07711-7 · Cited by 172 · see citing works
“On the other hand, PPO augmented with”
A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160 · see citing works
A survey on model-based reinforcement learning
Luo, F.-M., Xu, T., Lai, H., Chen, X., Zhang, W., & Yu, Y. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3696-5 · Cited by 173 · see citing works
Reinforcement learning for robot research: A comprehensive review and open issues
Zhang, T., & Mo, H. (2021) · International Journal of Advanced Robotic Systems · doi:10.1177/17298814211007305 · Cited by 150 · see citing works
Transferring policy of deep reinforcement learning from simulation to reality for robotics
Ju, H., Juan, R., Gomez, R., Nakamura, K., & Li, G. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00573-6 · Cited by 147 · see citing works
Hu, X.-M., Li, S., Huang, T., Tang, B., Huai, R., & Chen, L. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3312777 · Cited by 173 · see citing works
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128 · see citing works
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
Meta-Learning without MemorizationPreprint
Yin, M., Tucker, G., Zhou, M., Levine, S., & Finn, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.03820 · Cited by 113 · see citing works
Self-citationDeep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey
Majid, A. Y., Majid, A. Y., Saaybi, S., van Rietbergen, T., François-Lavet, V., Prasad, R. R. V., & Verhoeven, C. J. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264540 · Cited by 108 · see citing works
Meta reinforcement learning for sim-to-real domain adaptation
Arndt, K., Hazara, M., Ghadirzadeh, A., & Kyrki, V. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196540 · Cited by 97 · see citing works
Accelerating Convergence of Federated Learning in MEC with Dynamic Community
Sun, W. Y., Zhao, Y., Ma, W., Guo, B., Xu, L., & Duong, T. Q. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3241770 · Cited by 93 · see citing works
Tonioni, A., Rahnama, O., Joy, T., Stefano, L. D., Ajanthan, T., & Torr, P. H. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00989 · Cited by 84 · see citing works
Learning Fast Adaptation With Meta Strategy Optimization
Yu, W., Tan, J., Bai, Y., Coumans, E., & Ha, S. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974685 · Cited by 86 · see citing works
A Review of Safe Reinforcement Learning Methods for Modern Power Systems
Su, T., Wu, T., Zhao, J., Scaglione, A., & Xie, L. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3584656 · Cited by 91 · see citing works
Self-Supervised Deep Visual Odometry With Online Adaptation
Li, S., Wang, X., Cao, Y., Xue, F., Yan, Z., & Zha, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00637 · Cited by 78 · see citing works
Meta-seg: A survey of meta-learning for image segmentation
Luo, S., Li, Y., Gao, P., Wang, Y., & Serikawa, S. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108586 · Cited by 79 · see citing works
Learning quadrotor dynamics for precise, safe, and agile flight control
Saviolo, A., & Loianno, G. (2023) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2023.03.009 · Cited by 76 · see citing works
Yoo, A. H., & Collins, A. G. E. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01808 · Cited by 73 · see citing works
Adapting Rapid Motor Adaptation for Bipedal Robots
Kumar, A., Li, Z., Zeng, J., Pathak, D., Sreenath, K., & Malik, J. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981091 · Cited by 73 · see citing works
Fusion of Machine Learning and MPC under Uncertainty: What Advances Are on the Horizon?
Mesbah, A., Wabersich, K. P., Schoellig, A. P., Zeilinger, M. N., Lucia, S., Badgwell, T. A., & Paulson, J. A. (2022) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc53348.2022.9867643 · Cited by 70 · see citing works
Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
Song, X., Yang, Y., Choromański, K., Caluwaerts, K., Gao, W., Finn, C., & Tan, J. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341571 · Cited by 67 · see citing works
Self-citationMeta-Reinforcement Learning in Non-Stationary and Dynamic Environments
Bing, Z., Lerch, D., Huang, K., & Knoll, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3185549 · Cited by 67 · see citing works
Unsupervised Meta-Learning for Reinforcement LearningPreprint
Gupta, A., Eysenbach, B., Finn, C., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04640 · Cited by 66 · see citing works
Self-citationChen, L., Meng, F., & Zhang, Y. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3164023 · Cited by 65 · see citing works
Wu, J., Ye, Y., & Du, J. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105253 · Cited by 69 · see citing works
Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning
Huang, R., Chen, Y., Yin, T., Huang, Q., Tan, J., Yu, W., Li, X., Li, A., & Du, Y. (2022) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2022.3155117 · Cited by 65 · see citing works
Lifelong Incremental Reinforcement Learning With Online Bayesian Inference
Wang, Z., Chen, C., & Dong, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3055499 · Cited by 62 · see citing works
Context-aware Dynamics Model for Generalization in Model-Based Reinforcement LearningPreprint
Lee, K., Seo, Y., Lee, S., Lee, H., & Shin, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06800 · Cited by 64 · see citing works
Wang, J., Hu, J. W., Min, G., Ni, Q., & El-Ghazawi, T. A. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3197706 · Cited by 66 · see citing works
Learning Compositional Koopman Operators for Model-Based ControlPreprint
Li, Y., He, H., Wu, J., Katabi, D., & Torralba, A. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08264 · Cited by 61 · see citing works
Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems
Richards, S. M., Azizan, N., Slotine, J.-J. E., & Pavone, M. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.056 · Cited by 60 · see citing works
SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
Jiang, Y., Zhang, T., Ho, D., Bai, Y., Liu, C. K., Levine, S., & Tan, J. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561731 · Cited by 56 · see citing works
Self-citationActive Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control
Saviolo, A., Frey, J., Rathod, A., Diehl, M., & Loianno, G. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3339543 · Cited by 56 · see citing works
Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance
Luo, S., Androwis, G. J., Adamovich, S., Su, H., Nunez, E. H., & Zhou, X. A. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.702845 · Cited by 55 · see citing works
Wanna Make Your TCP Scheme Great for Cellular Networks? Let Machines Do It for You!
Abbasloo, S., Yen, C.-Y., & Chao, H. J. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036958 · Cited by 61 · see citing works
Bayesian Multi-Task Learning MPC for Robotic Mobile Manipulation
Arcari, E., Minniti, M. V., Scampicchio, A., Carron, A., Farshidian, F., Hutter, M., & Zeilinger, M. N. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3264758 · Cited by 52 · see citing works
Small Sample Learning in Big Data EraPreprint
Shu, J., Xu, Z., & Meng, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04572 · Cited by 78 · see citing works
Coupling Vision and Proprioception for Navigation of Legged Robots
Fu, Z., Kumar, A., Agarwal, A., Qi, H., Malik, J., & Pathak, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01676 · Cited by 51 · see citing works
Challenges of Reinforcement Learning
Ding, Z., & Dong, H. (2020) · Springer eBooks · doi:10.1007/978-981-15-4095-0_7 · Cited by 52 · see citing works
Safe Active Dynamics Learning and Control: A Sequential Exploration–Exploitation Framework
Lew, T., Sharma, A., Harrison, J., Bylard, A., & Pavone, M. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3154715 · Cited by 50 · see citing works
Virtual to Real-World Transfer Learning: A Systematic Review
Ranaweera, M., & Mahmoud, Q. H. (2021) · Electronics · doi:10.3390/electronics10121491 · Cited by 48 · see citing works
Seo, G., Yoon, S., Kim, M., Mun, C., & Hwang, E. (2021) · IEEE Access · doi:10.1109/access.2021.3094466 · Cited by 47 · see citing works
Massively Digitized Power Grid: Opportunities and Challenges of Use-Inspired AI
Xie, L., Zheng, X., Sun, Y., Huang, T., & Bruton, T. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3175070 · Cited by 52 · see citing works
Chen, Z., Zhang, L., Cai, W., Laili, Y., Wang, X., Wang, F., & Wang, H. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.01.010 · Cited by 44 · see citing works
What Can Learned Intrinsic Rewards Capture?Preprint
Zheng, Z., Oh, J., Hessel, M., Xu, Z., Kroiss, M., van Hasselt, H. P., Silver, D., & Singh, S. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05500 · Cited by 43 · see citing works
Federated Ensemble Model-Based Reinforcement Learning in Edge Computing
Wang, J., Hu, J. W., Mills, J., Min, G., Xia, M., & Georgalas, N. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3264480 · Cited by 41 · see citing works
Continual Reinforcement Learning in 3D Non-stationary Environments
Lomonaco, V., Desai, K., Culurciello, E., & Maltoni, D. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00132 · Cited by 36 · see citing works
Deep reinforcement learning towards real-world dynamic thermal management of data centers
Zhang, Q., Zeng, W., Lin, Q., Chng, C.-B., Chui, C., & Lee, P. S. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.120561 · Cited by 44 · see citing works
MetaABR: A Meta-Learning Approach on Adaptative Bitrate Selection for Video Streaming
Li, W., Li, X., Xu, Y., Yi, Y., & Lu, S. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3260086 · Cited by 40 · see citing works
Learning in Volatile Environments With the Bayes Factor Surprise
Liakoni, V., Modirshanechi, A., Wulfram Gerstner, B., & Brea, J. (2021) · Neural Computation · doi:10.1162/neco_a_01352 · Cited by 36 · see citing works
Wabersich, K. P., & Zeilinger, M. N. (2018) · arXiv (Cornell University) · Cited by 37 · see citing works
Meta Reinforcement Learning-Based Lane Change Strategy for Autonomous Vehicles
Ye, F., Wang, P., Chan, C., & Zhang, J. (2021) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv48863.2021.9575379 · Cited by 36 · see citing works
learn2learn: A Library for Meta-Learning ResearchPreprint
Arnold, S. M. R., Mahajan, P., Datta, D., Bunner, I., & Saitas-Zarkias, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12284 · Cited by 36 · see citing works
Chen, L., Wang, W., Liu, Z., Lai, C., & Zhou, B. (2019) · IEEE Access · doi:10.1109/access.2019.2909579 · Cited by 36 · see citing works
Interactron: Embodied Adaptive Object Detection
Kotar, K., & Mottaghi, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01444 · Cited by 35 · see citing works
How to Train Your MAML to Excel in Few-Shot ClassificationPreprint
Ye, H.-J., & Chao, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.16245 · Cited by 35 · see citing works
Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot TasksPreprint
Goldblum, M., Reich, S., Fowl, L., Ni, R., Cherepanova, V., & Goldstein, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06753 · Cited by 34 · see citing works
A Comprehensive Overview and Survey of Recent Advances in Meta-LearningPreprint
Peng, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11149 · Cited by 33 · see citing works
Image-based Deep Reinforcement Learning for Autonomous Lunar Landing
Scorsoglio, A., Furfaro, R., Linares, R., & Gaudet, B. (2020) · AIAA Scitech 2020 Forum · doi:10.2514/6.2020-1910 · Cited by 31 · see citing works
Continuous Meta-Learning without TasksPreprint
Harrison, J., Sharma, A., Finn, C., & Pavone, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.08866 · Cited by 31 · see citing works
Self-citationDynamic Navigation in Unconstrained Environments Using Reinforcement Learning Algorithms
Chronis, C., Anagnostopoulos, G., Politi, E., Dimitrakopoulos, G. J., & Varlamis, I. (2023) · IEEE Access · doi:10.1109/access.2023.3326435 · Cited by 33 · see citing works
Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
Chen, B., Liu, Z., Zhu, J., Xu, M., Ding, W., Li, L., & Zhao, D. X. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561593 · Cited by 30 · see citing works
Cheng, Y.-K., Zhao, P., Wang, F.-X., Block, D., & Hovakimyan, N. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3169309 · Cited by 12 · see citing works
Wang, H., Liu, Z., Han, Z., Wu, Y., & Liu, D. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3271900 · Cited by 31 · see citing works
Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platforms
Ghadirzadeh, A., Chen, X., Poklukar, P., Finn, C., Björkman, M., & Kragić, D. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636628 · Cited by 28 · see citing works
Self-citationSafe Reinforcement Learning via Statistical Model Predictive Shielding
Bastani, O., & Li, S. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.026 · Cited by 29 · see citing works
Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy
Luo, F.-M., Jiang, S., Yu, Y., Zhang, Z., & Zhang, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20730 · Cited by 28 · see citing works
Landmark-Guided Subgoal Generation in Hierarchical Reinforcement LearningPreprint
Kim, J., Seo, Y., & Shin, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13625 · Cited by 27 · see citing works
A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning
Wang, Z., Chen, C., & Dong, D. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3170485 · Cited by 25 · see citing works
Offline Meta-Reinforcement Learning with Advantage WeightingPreprint
Mitchell, E., Rafailov, R., Peng, X. B., Levine, S., & Finn, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06043 · Cited by 26 · see citing works
Self-citationCheng, J., Cheng, M., Liu, Y., Wu, J., Li, W., & Frangopol, D. M. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110127 · Cited by 25 · see citing works
Fast Task Adaptation Based on the Combination of Model-Based and Gradient-Based Meta Learning
Xu, Z., Chen, X., & Cao, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3028378 · Cited by 25 · see citing works
Brüggemann, S., & Possieri, C. (2020) · IEEE Control Systems Letters · doi:10.1109/lcsys.2020.3032083 · Cited by 25 · see citing works
Meta-Adaptive Nonlinear Control: Theory and Algorithms
Shi, G., Azizzadenesheli, K., O'Connell, M., Chung, S., & Yue, Y. (2021) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2106.06098 · Cited by 25 · see citing works
Meta-Reinforcement Learning in Nonstationary and Nonparametric Environments
Bing, Z., Knak, L., Cheng, L., Morin, F. O., Huang, K., & Knoll, A. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270298 · Cited by 28 · see citing works
PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement LearningPreprint
Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04152 · Cited by 23 · see citing works
Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformer
Lai, H., Zhang, W., He, X., Chen, Y., Tian, Z., Yu, Y. H., & Wang, J. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160497 · Cited by 22 · see citing works
Exploration by Maximizing Renyi Entropy for Reward-Free RL Framework
Zhang, C., Cai, Y., Huang, L., & Li, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17297 · Cited by 24 · see citing works
One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLPreprint
Kumar, S., Kumar, A., Levine, S., & Finn, C. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2010.14484 · Cited by 112 · see citing works
Self-citationRMA: Rapid Motor Adaptation for Legged RobotsPreprint
Kumar, A., Fu, Z., Pathak, D., & Malik, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04034 · Cited by 54 · see citing works
Adaptive meta-reinforcement learning for AUVs 3D guidance and control under unknown ocean currents
Jiang, Y. L., Zhang, K., Zhao, M., & Qin, H. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.118498 · Cited by 20 · see citing works
When Meta-Learning Meets Online and Continual Learning: A Survey
Son, J., Lee, S., & Kim, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3463709 · Cited by 24 · see citing works
- Authors
- Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1803.11347
- OpenAlex
- W2903630557
- Cited by
- 370
- References
- 55
- Citations by year
- 2026: 3 · 2025: 25 · 2024: 47 · 2023: 46 · 2022: 61 · 2021: 100 · 2020: 65 · 2019: 18 · 2018: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Biomedical Engineering › Robotic Locomotion and Control
- Keywords
- meta-reinforcement learning, online adaptation, model-based reinforcement learning, meta-learning, dynamics model, sample efficiency, continuous control, legged robots, millirobots, terrain adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
