The DOI matches its DataCite record; 53 of the 55 references it cites were found in a registry or an index.
Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
Open accessAbstract
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the problem settings in these papers were offline as they had two separate phases, one for learning and the other for evaluation.”
From Loss of plasticity in deep continual learning (2024) · cites this paper as [122]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Åström, K. J., & Wittenmark, B. (1989) · Cited by 5,009
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Self-citationHuman-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817
Optimization as a Model for Few-Shot Learning
Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766
End-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 2,641
Self-citationLearning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & de Freitas, N. (2016) · arXiv · doi:10.48550/arxiv.1606.04474 · Cited by 1,486
Research works printed but not linked: 6
- B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 , 2016. · arXiv
- K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114 , 2018. · arXiv
- M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foundations and Trends R © in Robotics , 2(1-2):1-142, 2013.
- Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning. CoRR , abs/1611.02779, 2016. · arXiv
- C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR , abs/1703.03400, 2017. · arXiv
- T. Munkhdalai and H. Yu. Meta networks. arXiv preprint arXiv:1703.00837 , 2017. · arXiv
Cited by
Sorted by citations
Items 301–371 of 371
Novel Adaptive Intelligent Control System Design
Duanyai, W., Song, W. K., Ka, M., Lee, D.-W., & Dissanayaka, S. (2025) · Electronics · doi:10.3390/electronics14153157 · Cited by 1
AI-driven quadruped robots: from fundamental locomotion to advanced biomimetic behaviors
Wu, L., Chow, C., Wong, W. R., Chuah, J. H., Kanesan, J., & Zhang, M. (2026) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2026.1855550 · Cited by 1
Shengliang, D., Guan, X., Sun, Z., Zhao, S., Shen, T., Chen, X., Duan, T., Wang, Y., Pan, J., Wu, Y., et al. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3140400 · Cited by 1
Safe adaptation in multiagent competition
Shen, M., & How, J. P. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981763 · Cited by 1
DAGA: Dynamics Aware Reinforcement Learning With Graph-Based Rapid Adaptation
Ji, J., Nie, B., & Gao, Y. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3249632 · Cited by 1
A Novel Framework for Adaptive Quadruped Robot Locomotion Learning in Uncertain Environments
Li, M., Guo, B., Zhao, K., Xu, R., Liu, S., Mao, S., Zhou, S., Xu, Q., & Yu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-99-9896-8_10 · Cited by 1
Self-citationApplying Machine learning to Performance Management: Obstacles and Potential
Kumar, P. (2024) · GBS Impact Journal of Multi Disciplinary Research · doi:10.58419/gbs.v9i2.922304 · Cited by 1
Fast Context Adaptation in Cost-Aware Continual Learning
Lahmer, S., Mason, F., Chiariotti, F., & Zanella, A. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3386647 · Cited by 1
Uncertainty-Aware Robot Control via Model-Based Reinforcement Learning
Zhou, Z. (2024) · International Conference on Neural Networks, Information and Communication Engineering (NNICE) · doi:10.1109/nnice61279.2024.10498155 · Cited by 1
Self-learning UAV Motion Planning Based on Meta Reinforcement Learning
Wang, M., Jiang, B., & Xie, J. (2024) · International Conference on Electronic Technology and Information Science (ICETIS) · doi:10.1109/icetis61828.2024.10593674 · Cited by 1
Lifelong Learning of Factored Policies via Policy Gradients
Mendez, J. A., & Eaton, E. (2020) · Cited by 1
Non-convex Optimization Using Parameter Continuation Methods for Deep Neural Networks
Pathak, H. N., & Paffenroth, R. C. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-981-15-6759-9_12 · Cited by 1
HyperDynamics: Generating Expert Dynamics Models by Observation
Zhou, X., Lal, S., Tung, H.-Y. F., Platanios, E. A., & Fragkiadaki, K. (2021) · International Conference on Learning Representations · Cited by 1
Domain Curiosity: Learning Efficient Data Collection Strategies for Domain Adaptation
Arndt, K., Struckmeier, O., & Kyrki, V. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635864 · Cited by 1
Zou, L. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-89931-4.00011-0 · Cited by 2
Liu, D., Wu, H., Sun, K., & Guan, Y. (2022) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio55434.2022.10011860 · Cited by 1
Local Nonparametric Meta-LearningPreprint
Goo, W., & Niekum, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03272 · Cited by 1
Technical Report: Adaptive Control for Linearizable Systems Using On-Policy Reinforcement LearningPreprint
Westenbroek, T., Mazumdar, E., Fridovich-Keil, D., Prabhu, V., Tomlin, C. J., & Sastry, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02766 · Cited by 1
System Identification via Meta-Learning in Linear Time-Varying EnvironmentsPreprint
Lin, S., Wang, H., & Zhang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14664 · Cited by 1
Achieving Effective Resource Management in Distributed SDN Controller Architectures
Huang, G. (2020) · doi:10.26686/wgtn.17151518.v1 · Cited by 1
Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and EnvironmentsPreprint
Anne, T., Wilkinson, J., & Li, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.07599 · Cited by 3
Learning Robust Controllers Via Probabilistic Model-Based Policy Search
Charvet, V., Jensen, B. S., & Murray-Smith, R. (2021) · arXiv · doi:10.48550/arxiv.2110.13576 · Cited by 1
GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RLPreprint
Sontakke, S., Iota, S., Hu, Z., Mehrjou, A., Itti, L., & Schölkopf, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15489 · Cited by 2
Absraction for Efficient Reinforcement Learning
Telfar, A. (2020) · doi:10.26686/wgtn.17147651 · Cited by 3
Bio-Inspired Adaptive Control of Robotic Manipulators for Grasping in Orbital Space Missions
Ogundipe, C. (2024) · doi:10.22215/etd/2024-15987 · Cited by 1
Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization
Röder, F., Benad, J., Eppe, M., & Banerjee, P. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-2438
Learning Shared Dynamics with Meta-World ModelsPreprint
Wu, L., Li, M., & Wang, Y.-X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01741
Time-Variant Variational Transfer for Value FunctionsPreprint
Canonaco, G., Soprani, A., Roveri, M., & Restelli, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.12864
Exploration by Maximizing Rényi Entropy for Zero-Shot Meta RL.Preprint
Zhang, C., Cai, Y., Huang, L., & Li, J. (2020) · arXiv (Cornell University)
Performance-Weighed Policy Sampling for Meta-Reinforcement LearningPreprint
Ahmed, I. H., Quiñones-Grueiro, M., & Biswas, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06016
Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNsPreprint
Ulmer, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00674
A Matching Network Focusing on the Relation between Samples
Zhao, J. L., & Gao, F. (2020) · IEEE International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA) · doi:10.1109/iciba50161.2020.9276759
Transfer Reinforcement Learning across Homotopy ClassesPreprint
Cao, Z., Kwon, M., & Sadigh, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05207
POLA: Online Time Series Prediction by Adaptive Learning RatesPreprint
Zhang, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08907
Robust Reinforcement Learning under model misspecificationPreprint
Yu, L., Wang, J., & Zhang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15370
Robustifying Reinforcement Learning Policies with L1 Adaptive ControlPreprint
Cheng, Y.-K., Zhao, P., Gandhi, M. S., Li, B., Theodorou, E. A., & Hovakimyan, N. (2021) · arXiv.org · doi:10.48550/arxiv.2106.02249
Model-Advantage Optimization for Model-Based Reinforcement Learning.Preprint
Modhe, N., Kamath, H. K., Batra, D., & Kalyan, A. (2021) · arXiv (Cornell University)
Shallow Representation is Deep: Learning Uncertainty-aware and Worst-case Random Feature DynamicsPreprint
Agudelo-España, D., Nemmour, Y., Schölkopf, B., & Zhu, J.-J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13066
Robustifying Reinforcement Learning Policies with L 1 Adaptive Control.Preprint
Cheng, Y., Zhao, P., Gandhi, M. S., Li, B., Theodorou, E. A., & Hovakimyan, N. (2021) · arXiv (Cornell University)
Out-of-distribution generalization of internal models is correlated with reward
Mann, K. S., Schneider, S., Chiappa, A. S., Lee, J. H., Bethge, M., Mathis, A., & Mathis, M. W. (2021) · International Conference on Learning Representations
Adaptation of Quadruped Robot Locomotion with Meta-LearningPreprint
Kuzhamuratov, A., Sorokin, D., Ulanov, A. E., & Lvovsky, A. I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03741
Brüggemann, S., & Possieri, C. (2021) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc50511.2021.9483344
Inter-Task Similarity for Lifelong Reinforcement Learning in Heterogeneous Tasks
Serrano, S. A. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/689
Inverse Reinforcement Learning with Expert Data in Multiple Environments
Arai, S. (2021) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.39.625
Learning to Cooperate with Unseen Agent via Meta-Reinforcement LearningPreprint
Charakorn, R., Manoonpong, P., & Dilokthanakul, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03431 · Cited by 2
Artificial intelligence-controlled pole balancing using an Arduino board
Orellana, J. L. R., & Chang, O. (2021) · Revista Tecnológica - ESPOL · doi:10.37815/rte.v33n2.852
Yadav, P., Kim, T., Suk, H., Lee, J., Jeong, H., & Kim, S. (2021) · doi:10.36227/techrxiv.17197913.v1
Yadav, P., Kim, T., Suk, H., Lee, J., Jeong, H., & Kim, S. (2021) · doi:10.36227/techrxiv.17197913
Transfer Dynamics in Emergent Evolutionary Curricula
Dharna, A., Hoover, A. K., Togelius, J., & Soros, L. B. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3151025 · Cited by 1
Meta-Conv1D Energy-Aware Path Planner for Mobile Robots in Unstructured TerrainsPreprint
Visca, M., Powell, R., Gao, Y., & Fallah, S. (2022) · doi:10.36227/techrxiv.19538575
Wang, T. (2022) · International Conference on Intelligent Computing and Control Systems (ICICCS) · doi:10.1109/iciccs53718.2022.9788169
RetractedAccelerating Robot Reinforcement Learning with Samples of Different Simulation Precision
Zhao, Y. B., Zhai, Y., Luo, J., Feng, D., Ding, B., & Li, Z. (2021) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys53884.2021.00080
A Novel Meta Control Framework for Robot Arm Reaching with Changeable Configuration
Hu, W., Yuan, Y., Wang, Y., & Luo, D. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309594
Chen, C., Li, C., Xiong, R., Gao, H., & Wang, Y. (2023) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus58632.2023.10318421
Guo, Q., Lian, R., Ma, K., Hou, L., & Wu, Y. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394355
Meta-Learning With Versatile Loss Geometries for Fast Adaptation Using Mirror Descent
Zhang, Y., Li, B., & Giannakis, G. B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448144
Constrained deep reinforcement learning for maritime platform defense
Markowitz, J. (2024) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.3014007
Dynamics Adaptive Safe Reinforcement Learning with a Misspecified Simulator
Xue, R., Zhang, Z., Li, L., Chen, F., Li, Y.-C., Yu, Y., & Yuan, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70368-3_5
Efficient Adversarially Guided Actor–Critic
Xu, M., Ge, S. S., Zhao, D., & Zhao, Q. (2024) · IEEE Transactions on Games · doi:10.1109/tg.2024.3453444
Patil, P., Agrawal, R., Thomas, S., Thejaswi, A., & Chandar, T. S. (2024) · International Conference on Robotics and Automation Sciences (ICRAS) · doi:10.1109/icras62427.2024.10654466
Model-free PAC Time-Optimal Control Synthesis with Reinforcement Learning
Liu, M., Lu, P., Chen, X., Sokolsky, O. V., Lee, I., & Kong, F. (2024) · ACM/IEEE International Conference on Formal Methods and Models for Co-Design (MEMOCODE) · doi:10.1109/memocode63347.2024.00009
Ho, J., Wang, C., King, C., Feng, C.-W., & Sunkuan, S.-Y. (2024) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc63610.2024.00024
Neural Network based Identification of 3-Φ Battery Energy Storage Systems with Different Capacities
Yadav, A., Kishor, N., Negi, R., & Lazzari, R. (2025) · International Conference on Power, Control and Computing Technologies (ICPC2T) · doi:10.1109/icpc2t63847.2025.10958657
Solving Sokoban using Hierarchical Reinforcement Learning with LandmarksPreprint
Pastukhov, S. (2025) · Qeios · doi:10.32388/uwvrik
Enerdast: a Free Energy Based Mechanism for Strategy Tuning in Sparse Reward Scenarios
Li, Z., Lang, Y., Wang, S., & Ren, Q. (2025) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc65474.2025.11090349
A study of the efficacy of generative flow networks for robotics and machine fault-adaptation
Sufiyan, Z., Golestan, S., Miwa, S., Mitsuka, Y., & Zai͏̈ane, O. R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111904
Reinforcement Learning in Hydropower: Pump Start-Up Control in a Simulated Environment
Tubeuf, C., Bernhardt, T., der Schmitten, J. A., Heitzinger, C., Birkelbach, F., Hofmann, R., & Maly, A. (2025) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/1561/1/012035
Kim, J., & Kang, D. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3688
Sathiya, A., Sridevi, A., Samuel Manoharan, J., & Sangeetha, A. (2026) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9798898815011126010007
Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization
Röder, F., Benad, J., Eppe, M., & Banerjee, P. K. (2025) · arXiv
Metadata
- Authors
- Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1803.11347
- OpenAlex
- W2903630557
- Cited by
- 370
- References
- 55
- Citations by year
- 2026: 3 · 2025: 25 · 2024: 47 · 2023: 46 · 2022: 61 · 2021: 100 · 2020: 65 · 2019: 18 · 2018: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotic Locomotion and Control
- Keywords
- meta-reinforcement learning, online adaptation, model-based reinforcement learning, meta-learning, dynamics model, sample efficiency, continuous control, legged robots, millirobots, terrain adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
