Sim-to-Real in Reinforcement Learning for Everyone
Abstract
In reinforcement learning (RL), it remains a challenge to have a robotic agent perform a task in the real world for which it was trained in simulation. In this paper, we present our work training a low-cost robotic arm in simulation to move towards a predefined target in space, represented by a red ball in an RGB image, and transferring the capability to the real arm. We exercised the entire end-to-end flow including the 3D modeling of the arm, training of a state-of-the-art RL policy in simulation with multiple actors in a distributed fashion, domain randomization in order to close the sim-to-real gap, and finally the execution of the trained model in the real robot. We also implemented a mechanism to edit the image captured from the camera before sending it to the model for inference, which allowed us to automate reward computation in the physical world. Our work highlights important challenges of training RL agents and moving them to the real world, validating important aspects shown by other works as well as detailing steps not explained by some of them (e.g. how to compute the reward in the real world). The conducted experiments show the improvements observed as the techniques were added to the final solution.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Domain randomization and dynamics randomization methods focus on introducing perturbations in simulation environments with the aim of making agents less susceptible to mismatches between simulation and reality.
“Domain randomization and dynamics randomization methods focus on introducing perturbations in the simulation environments with the aim of making the agents less susceptible to the mismatches between simulation and reality [30], [38], [40].”
From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [38]
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,821 · see citing works
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733 · see citing works
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 3× · see citing works
“Domain randomization is a set of techniques to reduce the differences between the simulation environment and the real world [9].”
BackgroundMethodInfluentialDeep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Unity: A General Platform for Intelligent Agents
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.02627 · Cited by 626 · cited 1× · see citing works
“In addition, it contains Unity ML-Agents Toolkit, which enables the training of intelligent agents [12].”
BackgroundOpenAI GymPreprint
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv (Cornell University) · Cited by 544 · see citing works
James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420 · see citing works
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Zhao, W., Queralta, J. P., & Westerlund, T. (2020) · 2020 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci47803.2020.9308468 · Cited by 941 · see citing works
- Authors
- Vacaro, J., Marques, G., Oliveira, B., Paz, G., Paula, T., Staehler, W., & Murphy, D.
- Affiliations
- Hewlett-Packard (United States) (US)
- Venue
- 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 305-310
- OpenAlex
- W3009584224
- Cited by
- 13
- References
- 12
- Citations by year
- 2025: 4 · 2024: 3 · 2023: 5 · 2020: 1
- FWCI
- 0.10
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Latin American Robotics Symposium, Brazilian Symposium on Robotics and Workshop on Robotics in Education (LARS/SBR/WRE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- sim-to-real transfer, reinforcement learning, robot manipulators, domain randomization, low-cost robot, distributed training, end-to-end control
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
