The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience
Open accessAbstract
We consider the problem of transferring policies to the real world by training on a distribution of simulated scenarios. Rather than manually tuning the randomization of simulations, we adapt the simulation parameter distribution using a few real world roll-outs interleaved with policy training. In doing so, we are able to change the distribution of simulations to improve the policy transfer by matching the policy behavior in simulation and the real world. We show that policies trained with our method are able to reliably transfer to different robots in two real world tasks: swing-peg-in-hole and opening a cabinet drawer. The video of our experiments can be found at https://sites.google.com/view/simopt.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
System Identification: Theory for the User
Lennart Ljung, A. (1987) · Cited by 17,690
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 5×
“We noticed that the RL training can be sped up by initializing the policy with the weights from the previous SimOpt iteration, effectively reducing the number of needed PPO iterations from 200 to 10 after the first SimOpt iteration.”
MethodInfluentialDomain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 3×
“As it was shown in [6, 7, 9], it is possible to design a distribution of simulation parameters p (⇠), such that a policy trained on P⇠⇠p would perform well on a real world dynamics distribution.”
BackgroundMethodLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
Learning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · cited 1×
“[22] use randomization of physical parameters of the scene to learn a policy in simulation and transfer it to a real robot for pushing a puck to a target position.”
Simultaneous Deep Transfer Across Domains and Tasks
Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.463 · Cited by 1,149
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460875 · Cited by 636
Noise and the reality gap: The use of simulation in evolutionary robotics
Jakobi, N., Husbands, P., & Harvey, I. (1995) · Lecture Notes in Computer Science · doi:10.1007/3-540-59496-5_337 · Cited by 631
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618
Block-oriented Nonlinear System Identification
(2010) · Lecture Notes in Control and Information Sciences · doi:10.1007/978-1-84996-513-2 · Cited by 608
Time-Contrastive Networks: Self-Supervised Learning from Video
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., & Brain, G. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8462891 · Cited by 546
Self-citationRelative Entropy Policy Search
Peters, J., Mulling, K., & Altun, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 455
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · cited 1×
“resentations of real and synthetic data [16], using generative models to make synthetic images more realistic [17], fine-tuning the policies trained in simulation in the real world [18], learning inverse dynamics models [19], multiobjective optimization of task fitness and transferability to the real world [20], train…”
MethodEPOpt: Learning Robust Neural Network Policies Using Model EnsemblesPreprint
Rajeswaran, A., Ghotra, S., Ravindran, B., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01283 · Cited by 373
Asymmetric Actor Critic for Image-Based Robot Learning
Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., & Abbeel, P. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.008 · Cited by 343
Learning Dexterous In-Hand Manipulation
OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., et al. (2018) · arXiv · doi:10.48550/arxiv.1808.00177 · Cited by 248
Preparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237
Preparing for the Unknown: Learning a Universal Policy with Online System Identification
Yu, W., Tan, J., Liu, C. K., & Turk, G. (2017) · arXiv · doi:10.48550/arxiv.1702.02453 · Cited by 237
Simultaneous Deep Transfer Across Domains and Tasks
Hoffman, J., Tzeng, E., Darrell, T., & Saenko, K. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_9 · Cited by 210
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
James, S., Davison, A. J., & Johns, E. (2017) · arXiv · doi:10.48550/arxiv.1707.02267 · Cited by 208
Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
Deisenroth, M., Rasmussen, C., & Fox, D. (2011) · Robotics: Science and Systems VII · doi:10.15607/rss.2011.vii.008 · Cited by 205
Self-citationTransfer from Simulation to Real World through Learning Deep Inverse Dynamics ModelPreprint
Christiano, P. F., Shah, Z., Mordatch, I., Schneider, J., Blackwell, T., Tobin, J., Abbeel, P., & Zaremba, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03518 · Cited by 159
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial NetsPreprint
Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G. S., & Lim, J. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10479 · Cited by 157
Self-citationDART: Dense Articulated Real-Time Tracking
Schmidt, T., Newcombe, R. A., & Fox, D. (2014) · Robotics: Science and Systems · doi:10.15607/rss.2014.x.030 · Cited by 144
Self-citationCollective robot reinforcement learning with distributed asynchronous guided policy search
Yahya, A., Li, A., Kalakrishnan, M., Chebotar, Y., & Levine, S. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202141 · Cited by 143
Self-citationEnsemble-CIO: Full-body dynamic motion planning that transfers to physical humanoids
Mordatch, I., Lowrey, K., & Todorov, E. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354126 · Cited by 135
Crossing the reality gap in evolutionary robotics by promoting transferable controllers
Koos, S., Mouret, J.-B., & Doncieux, S. (2010) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1830483.1830505 · Cited by 116
Grounded Action Transformation for Robot Learning in Simulation
Hanna, J. P., & Stone, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11044 · Cited by 81
Reinforcement Learning for Pivoting TaskPreprint
Antonova, R., Cruciani, S., Smith, C., & Kragić, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00472 · Cited by 77
Humanoid robots learning to walk faster: from the real world to simulation and back
Farchy, A., Barrett, S., MacAlpine, P., & Stone, P. (2013) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/2484920.2484930 · Cited by 67
Towards Adapting Deep Visuomotor Representations from Simulated to Real Environments.Preprint
Tzeng, E., Devin, C., Hoffman, J., Finn, C., Peng, X., Levine, S., Saenko, K., & Darrell, T. (2015) · arXiv (Cornell University) · Cited by 66
Domain Randomization for Simulation-Based Policy Optimization with Transferability Assessment
Muratore, F., Treede, F., Gienger, M., & Peters, J. (2018) · MPG.PuRe (Max Planck Society) · Cited by 65
Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system
Lowrey, K., Kolev, S., Dao, J., Rajeswaran, A., & Todorov, E. V. (2018) · IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR) · doi:10.1109/simpar.2018.8376268 · Cited by 63
Physically consistent state estimation and system identification for contacts
Kolev, S., & Todorov, E. V. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363481 · Cited by 55
Fast Model Identification via Physics Engines for Data-Efficient Policy Search
Zhu, S., Kimmel, A. S., Bekris, K. E., & Boularias, A. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/451 · Cited by 53
Mutual Alignment Transfer LearningPreprint
Wulfmeier, M., Posner, I., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07907 · Cited by 49
Simulation-based design of dynamic controllers for humanoid balancing
Tan, J., Xie, Z., Boots, B., & Liu, C. K. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759424 · Cited by 43
Cited by
An Introduction to Deep Reinforcement Learning
Vincent, F.-L., Peter, H., Riashat, I., Marc, G. B., & Joelle, P. (2018) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000071 · Cited by 1,286
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
Solving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602
Synthetic Data for Deep Learning
Nikolenko, S. I. (2021) · Springer Optimization and Its Applications · doi:10.1007/978-3-030-75178-4 · Cited by 423
James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420
Metadata
- Authors
- Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., & Fox, D.
- Venue
- 2019 International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 8973-8979
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2897345632
- Cited by
- 457
- References
- 45
- FWCI
- 25.74
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.