The DOI matches its DataCite record; 53 of the 55 references it cites were found in a registry or an index.
Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
Open accessAbstract
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the problem settings in these papers were offline as they had two separate phases, one for learning and the other for evaluation.”
From Loss of plasticity in deep continual learning (2024) · cites this paper as [122]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Åström, K. J., & Wittenmark, B. (1989) · Cited by 5,009
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Self-citationHuman-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817
Optimization as a Model for Few-Shot Learning
Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766
End-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 2,641
Self-citationLearning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & de Freitas, N. (2016) · arXiv · doi:10.48550/arxiv.1606.04474 · Cited by 1,486
Adaptive control of linearizable systems
Sastry, S., & Isidori, A. (1989) · IEEE Transactions on Automatic Control · doi:10.1109/9.40741 · Cited by 1,195
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
A Simple Neural Attentive Meta-LearnerPreprint
Mishra, N., Rohaninejad, M., Chen, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03141 · Cited by 961
Self-citationNeural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8463189 · Cited by 898
Self-citationReinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Continuous Deep Q-Learning with Model-based AccelerationPreprint
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 770
Self-citationLearning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579
Learning to Learn: Introduction and Overview
Thrun, S., & Pratt, L. (1998) · Learning to Learn · doi:10.1007/978-1-4615-5529-2_1 · Cited by 521
One-shot Learning with Memory-Augmented Neural NetworksPreprint
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06065 · Cited by 515
Online Parameter Estimation and Adaptive Control of Permanent-Magnet Synchronous Machines
Underwood, S., & Husain, I. (2009) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2009.2036029 · Cited by 502
Information theoretic MPC for model-based reinforcement learning
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., & Theodorou, E. A. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989202 · Cited by 495
Levine, S., & Koltun, V. (2013) · Cited by 481
Self-citationModel-Ensemble Trust-Region Policy OptimizationPreprint
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10592 · Cited by 405
Self-citationLearning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.131 · Cited by 357
DeepMPC: Learning Deep Latent Features for Model Predictive Control
Lenz, I., Knepper, R. A., & Saxena, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.012 · Cited by 349
GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models
Ko, J., & Fox, D. (2009) · Autonomous Robots · doi:10.1007/s10514-009-9119-x · Cited by 342
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive EnvironmentsPreprint
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03641 · Cited by 303
Self-citationOnline movement adaptation based on previous sensor experiences
Pástor, P., Righetti, L., Kalakrishnan, M., & Stefan Schaal, A. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6095059 · Cited by 224
Finn, C., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11622 · Cited by 195
Self-citationMeta-neural networks that learn by learning
Naik, D., & Mammone, R. J. (2003) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.287172 · Cited by 185
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Lu, L., Murray, I., & Renals, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07959 · Cited by 170
Learning to OptimizePreprint
Li, K., & Malik, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01885 · Cited by 159
Bayesian Recurrent Neural Networks
Fortunato, M., Blundell, C., & Vinyals, O. (2017) · arXiv · doi:10.48550/arxiv.1704.02798 · Cited by 145
Learning to Learn: Meta-Critic Networks for Sample Efficient LearningPreprint
Sung, F., Zhang, L., Xiang, T., Hospedales, T. M., & Yang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09529 · Cited by 132
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors
Fu, J., Levine, S., & Abbeel, P. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759592 · Cited by 126
Self-citationLearning Optimal Adaptation Strategies in Unpredictable Motor Tasks
Braun, D. A., Aertsen, A., Wolpert, D. M., & Mehring, C. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.3075-08.2009 · Cited by 114
Meta Reinforcement Learning with Latent Variable Gaussian Processes
Sæmundsson, S., Hofmann, K., & Deisenroth, M. P. (2018) · arXiv · doi:10.48550/arxiv.1803.07551 · Cited by 106
Dynamic Evaluation of Neural Sequence Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2017) · arXiv · doi:10.48550/arxiv.1709.07432 · Cited by 90
Optimization of Perturbative PV MPPT Methods Through Online System Identification
Manganiello, P., Ricco, M., Petrone, G., Monmasson, É., & Spagnuolo, G. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2014.2317143 · Cited by 77
Model Predictive Path Integral Control using Covariance Variable Importance Sampling
Williams, G., Aldrich, A., & Theodorou, E. (2015) · arXiv · doi:10.48550/arxiv.1509.01149 · Cited by 71
LEARNING TO GENERATE ARTIFICIAL FOVEA TRAJECTORIES FOR TARGET DETECTION
Schmidhuber, J., & Huber, R. (1991) · International Journal of Neural Systems · doi:10.1142/s012906579100011x · Cited by 69
Aswani, A. J., Bouffard, P., & Tomlin, C. J. (2012) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2012.6315483 · Cited by 68
Meta-learning with backpropagation
Younger, A. S., Hochreiter, S., & Conwell, P. R. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2001.938471 · Cited by 66
Online System Identification and Adaptive Control for PEM Fuel Cell Maximum Efficiency Tracking
Kélouwani, S., Adegnon, K. M., Agbossou, K., & Dubé, Y. (2012) · IEEE Transactions on Energy Conversion · doi:10.1109/tec.2012.2194496 · Cited by 56
Learning feedback terms for reactive planning and control
Rai, A., Sutanto, G., Stefan Schaal, A., & Meier, F. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989252 · Cited by 50
Towards robust online inverse dynamics learning
Meier, F., Kappler, D., Ratliff, N., & Stefan Schaal, A. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759594 · Cited by 41
Adaptive model predictive control for constrained linear systems
Tanasković, M., Fagiano, L., Smith, R. S., Goulart, P. J., & Morari, M. (2013) · European Control Conference (ECC) · doi:10.23919/ecc.2013.6669544 · Cited by 38
Learning Image-Conditioned Dynamics Models for Control of Underactuated Legged Millirobots
Nagabandi, A., Yang, G., Asmar, T., Pandya, R., Kahn, G., Levine, S., & Fearing, R. S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594193 · Cited by 34
Self-citationModel-based policy search for automatic tuning of multivariate PID controllers
Doerr, A., Nguyen-Tuong, D., Marco, A., Stefan Schaal, A., & Trimpe, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989622 · Cited by 33
Drifting Gaussian processes with varying neighborhood sizes for online model learning
Meier, F., & Stefan Schaal, A. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487143 · Cited by 28
Structure Learning in Motor Control:A Deep Reinforcement Learning ModelPreprint
Weinstein, A., & Botvinick, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06827 · Cited by 12
Learning Rapid-Temporal AdaptationsPreprint
Munkhdalai, T., Yuan, X., Mehri, S., Wang, T., & Trischler, A. (2017) · arXiv (Cornell University) · Cited by 10
Online representation learning in recurrent neural language models
Rei, M. (2015) · doi:10.17863/cam.21361 · Cited by 6
Research works printed but not linked: 6
- B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 , 2016. · arXiv
- K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114 , 2018. · arXiv
- M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foundations and Trends R © in Robotics , 2(1-2):1-142, 2013.
- Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning. CoRR , abs/1611.02779, 2016. · arXiv
- C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR , abs/1703.03400, 2017. · arXiv
- T. Munkhdalai and H. Yu. Meta networks. arXiv preprint arXiv:1703.00837 , 2017. · arXiv
Cited by
Sorted by citations
Deep Reinforcement Learning: A Survey
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., & Miao, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3207346 · Cited by 972
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
Learning-Based Model Predictive Control: Toward Safe Learning in Control
Hewing, L., Wabersich, K. P., Menner, M., & Zeilinger, M. N. (2019) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-090419-075625 · Cited by 849
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Self-citationModel-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000086 · Cited by 871
Metadata
- Authors
- Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1803.11347
- OpenAlex
- W2903630557
- Cited by
- 370
- References
- 55
- Citations by year
- 2026: 3 · 2025: 25 · 2024: 47 · 2023: 46 · 2022: 61 · 2021: 100 · 2020: 65 · 2019: 18 · 2018: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotic Locomotion and Control
- Keywords
- meta-reinforcement learning, online adaptation, model-based reinforcement learning, meta-learning, dynamics model, sample efficiency, continuous control, legged robots, millirobots, terrain adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
