The DOI matches its DataCite record.
Meta Reinforcement Learning for Sim-to-real Domain Adaptation
Open accessAbstract
Modern reinforcement learning methods suffer from low sample efficiency and unsafe exploration, making it infeasible to train robotic policies entirely on real hardware. In this work, we propose to address the problem of sim-to-real domain transfer by using meta learning to train a policy that can adapt to a variety of dynamic conditions, and using a task-specific trajectory generation model to provide an action space that facilitates quick exploration. We evaluate the method by performing domain adaptation in simulation and analyzing the structure of the latent space during adaptation. We then deploy this policy on a KUKA LBR 4+ robot and evaluate its performance on a task of hitting a hockey puck to a target. Our method shows more consistent and stable domain adaptation than the baseline, resulting in better overall performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Some of the approaches to bridge the gap in this direction rely on meta learning [11] or continual learning [12], among others.”
From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 28
- V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning," arXiv preprint arXiv:1312.5602, 2013. · arXiv
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, p. 529, 2015.
- E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control," 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
- J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox, "Gpu-accelerated robotic simulation for distributed reinforce- ment learning," CoRR, vol. abs/1810.05762, 2018. · arXiv
- Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, "Closing the sim-to-real loop: Adapting simulation randomization with real world experience," arXiv preprint arXiv:1810.05687, 2018. · arXiv
- J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bo- hez, and V. Vanhoucke, "Sim-to-real: Learning agile locomotion for quadruped robots," arXiv preprint arXiv:1804.10332, 2018. · arXiv
- A. Hämäläinen, K. Arndt, A. Ghadirzadeh, and V. Kyrki, "Affordance learning for end-to-end visuomotor robot control," arXiv preprint arXiv:1903.04053, 2019. · arXiv
- X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, "Sim-to- real transfer of robotic control with dynamics randomization," in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8, IEEE, 2018.
- M. Hazara and V. Kyrki, "Transferring generalizable motor primitives from simulation to real world," IEEE Robotics and Automation Letters, vol. 4, pp. 2172-2179, April 2019.
- J. Schmidhuber, "Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meta...-hook," diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987.
- J. Schmidhuber, "A neural network that embeds its own meta-levels," in IEEE International Conference on Neural Networks, pp. 407-412 vol.1, March 1993.
- J. Schmidhuber, J. Zhao, and N. N. Schraudolph, "Learning to learn," ch. Reinforcement Learning with Self-modifying Policies, pp. 293- 309, Norwell, MA, USA: Kluwer Academic Publishers, 1998.
- S. Ravi and H. Larochelle, "Optimization as a model for few-shot learning," in ICLR, 2017.
- Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, "Rl2: Fast reinforcement learning via slow reinforcement learning," ArXiv, vol. abs/1611.02779, 2017. · arXiv
- T. Miconi, K. O. Stanley, and J. Clune, "Differentiable plasticity: train- ing plastic neural networks with backpropagation," in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 3556- 3565, 2018.
- C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in Proceedings of the 34th International Conference on Machine Learning (D. Precup and Y. W. Teh, eds.), vol. 70 of Proceedings of Machine Learning Research, (International Convention Centre, Sydney, Australia), pp. 1126-1135, PMLR, 06-11 Aug 2017.
- G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, "Openai gym," CoRR, vol. abs/1606.01540, 2016. · arXiv
- A. Antoniou, H. A. Edwards, and A. J. Storkey, "How to train your maml," ArXiv, vol. abs/1810.09502, 2018. · arXiv
- B. C. Stadie, G. Yang, R. Houthooft, X. Chen, Y. Duan, Y. Wu, P. Abbeel, and I. Sutskever, "Some considerations on learning to explore via meta-reinforcement learning," CoRR, vol. abs/1803.01118, 2018. · arXiv
- J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, "Domain randomization for transferring deep neural networks from simulation to the real world," in 2017 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. 23-30, IEEE, 2017.
- F. Sadeghi and S. Levine, "(cad)$ˆ2$rl: Real single-image flight without a single real image," CoRR, vol. abs/1611.04201, 2016. · arXiv
- F. Sadeghi, A. Toshev, E. Jang, and S. Levine, "Sim2real view invariant visual servoing by recurrent control," CoRR, vol. abs/1712.07642, 2017. · arXiv
- OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, "Learning dexterous in-hand manipulation," ArXiv, vol. abs/1808.00177, 2018. · arXiv
- V. Petrík and V. Kyrki, "Feedback-based fabric strip folding," CoRR, vol. abs/1904.01298, 2019. · arXiv
- M. Wulfmeier, I. Posner, and P. Abbeel, "Mutual alignment transfer learning," in 1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, pp. 281-290, 2017.
- I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel, "Model-based reinforcement learning via meta-policy op- timization," arXiv preprint arXiv:1809.05214, 2018. · arXiv
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," ArXiv, vol. abs/1707.06347, 2017. · arXiv
- S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," J. Mach. Learn. Res., vol. 17, pp. 1334- 1373, Jan. 2016.
Cited by
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941
Metadata
- Authors
- Arndt, K., Hazara, M., Ghadirzadeh, A., & Kyrki, V.
- Affiliations
- Aalto University (FI)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 2725-2731
- ISSN
- 1050-4729 (ISSN-L)
- arXiv
- 1909.12906
- OpenAlex
- W2976047560
- Cited by
- 98
- References
- 0
- Citations by year
- 2026: 7 · 2025: 23 · 2024: 11 · 2023: 20 · 2022: 13 · 2021: 11 · 2020: 12
- FWCI
- 5.27
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- meta-reinforcement learning, sim-to-real transfer, domain adaptation, trajectory generation, latent space structure, sample efficiency, safe exploration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
