CoChat
Sign Up
PreprintCited by 59Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record.

"Good Robot!": Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer

Open access
Hundt, Andrew, Killeen, Benjamin, Greene, Nicholas, Wu, Hongtao, Kwon, Heeyeon, Paxton, Chris, Hager, Gregory D.·2019·arXiv
DOI 10.48550/arxiv.1909.11730arXiv 1909.11730
Open this paper in CoChat — free accountOpen PDF

Abstract

Current Reinforcement Learning (RL) algorithms struggle with long-horizon tasks where time can be wasted exploring dead ends and task progress may be easily reversed. We develop the SPOT framework, which explores within action safety zones, learns about unsafe regions without exploring them, and prioritizes experiences that reverse earlier progress to learn with remarkable efficiency. The SPOT framework successfully completes simulated trials of a variety of tasks, improving a baseline trial success rate from 13% to 100% when stacking 4 cubes, from 13% to 99% when creating rows of 4 cubes, and from 84% to 95% when clearing toys arranged in adversarial patterns. Efficiency with respect to actions per trial typically improves by 30% or more, while training takes just 1-20k actions, depending on the task. Furthermore, we demonstrate direct sim to real transfer. We are able to create real stacks in 100% of trials with 61% efficiency and real rows in 100% of trials with 59% efficiency by directly loading the simulation-trained model on the real robot with no additional real-world fine-tuning. To our knowledge, this is the first instance of reinforcement learning with successful sim to real transfer applied to long term multi-step tasks such as block-stacking and row-making with consideration of progress reversal. Code is available at https://github.com/jhu-lcsr/good_robot .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1909.11730

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In cases where system identification for one-shot transfer is the objective, researchers have often built or customized specific simulations that meet problem-specific requirements and constraints.

    “In those cases where system identification for one-shot transfer is the objective, researchers have often built or customized specific simulations that meet problem-specific requirements and constraints [32], [36], [41].”

    From Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey (2020) · cites this paper as [41]

    The Simulation Experiments section (p. 5) describes two simulated tasks of the authors’ design, and the Real World Experiments section (p. 7) reports direct transfer of models trained in simulation. The paper does not describe system identification as the objective or establish that researchers often build or customize simulations to meet problem-specific constraints. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. [1][1] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser, 'Learning synergies between pushing and grasping with self-supervised deep reinforcement learning,' in IROS . IEEE, 2018, pp. 4238-4245. -
  2. [2][2] O. Kroemer, S. Niekum, and G. D. Konidaris, 'A review of robot learning for manipulation: Challenges, representations, and algorithms,' arXiv preprint arXiv:1907.03146 , 2019. - · arXiv
  3. [3][3] S. Levine, C. Finn, T. Darrell, and P. Abbeel, 'End-to-end training of deep visuomotor policies,' The Journal of Machine Learning Research , vol. 17, no. 1, pp. 1334-1373, 2016. -
  4. [4][4] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, 'Learning hand-eye coordination for robotic grasping with deep learning and largescale data collection,' The International Journal of Robotics Research , vol. 37, no. 4-5, pp. 421-436, 2018. -
  5. [5][5] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. , 'Qtopt: Scalable deep reinforcement learning for vision-based robotic manipulation,' CoRL , 2018. -
  6. [6][6] J. Redmon and A. Angelova, 'Real-time grasp detection using convolutional neural networks,' in ICRA . IEEE, 2015, pp. 1316-1322. -
  7. [7][7] B. Drost, M. Ulrich, N. Navab, and S. Ilic, 'Model globally, match locally: Efficient and robust 3d object recognition.' in CVPR , vol. 1, no. 2, 2010, p. 5. -
  8. [8][8] S. Kumra and C. Kanan, 'Robotic grasp detection using deep convolutional neural networks,' IROS , Sep 2017. -
  9. [9][9] E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz, and S. Levine, 'Endto-end learning of semantic grasping,' in CoRL , 2017, pp. 119-132. -
  10. [10][10] I. Lenz, H. Lee, and A. Saxena, 'Deep learning for detecting robotic grasps,' IJRR , vol. 34, no. 4-5, pp. 705-724, 2015. -
  11. [12][12] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox, '6-dof grasping for target-driven object manipulation in clutter,' ICRA , 2020. -
  12. [13][13] A. Gupta, C. Eppner, S. Levine, and P. Abbeel, 'Learning dexterous manipulation for a soft robotic hand from human demonstrations,' in IROS . IEEE, 2016, pp. 3786-3793. -
  13. [14][14] C. Devin, P. Abbeel, T. Darrell, and S. Levine, 'Deep object-centric representations for generalizable robot learning,' in ICRA . IEEE, 2018, pp. 7111-7118. -
  14. [15][15] A. Lerer, S. Gross, and R. Fergus, 'Learning physical intuition of block towers by example,' ICML , pp. 430-438, 2016. -
  15. [16][16] O. Groth, F. B. Fuchs, I. Posner, and A. Vedaldi, 'Shapestacks: Learning vision-based physical intuition for generalised object stacking,' in ECCV , 2018, pp. 702-717. -
  16. [17][17] C. Finn, I. J. Goodfellow, and S. Levine, 'Unsupervised learning for physical interaction through video prediction,' in Advances in Neural Information Processing Systems , 2016, pp. 64-72. -
  17. [18][18] A. Byravan and D. Fox, 'Se3-nets: Learning rigid body motion using deep neural networks,' in ICRA , 2017, pp. 173-180. -
  18. [20][20] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg, 'Learning ambidextrous robot grasping policies,' Science Robotics , vol. 4, no. 26, 2019. -
  19. [21][21] A. Mousavian, C. Eppner, and D. Fox, '6-dof graspnet: Variational grasp generation for object manipulation,' in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 2901-2910. -
  20. [22][22] F. Zhang, J. Leitner, M. Milford, and P. Corke, 'Modular deep q networks for sim-to-real transfer of visuo-motor policies,' ACRA , 2017. [Online]. Available: http://arxiv.org/abs/1610.06781 - · arXiv
  21. [23][23] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. A. Funkhouser, 'Tossingbot: Learning to throw arbitrary objects with residual physics,' in Robotics: Science and Systems XV , vol. 15, 2019. -
  22. [24][24] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, 'Domain randomization for transferring deep neural networks from simulation to the real world,' in IROS . IEEE, 2017, pp. 23-30. -
  23. [25][25] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, et al. , 'Using simulation and domain adaptation to improve efficiency of deep robotic grasping,' in ICRA . IEEE, 2018, pp. 4243-4250. -
  24. [26][26] Y. Zhu, Z. Wang, J. Merel, A. A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramr, R. Hadsell, N. de Freitas, and N. Heess, 'Reinforcement and imitation learning for diverse visuomotor skills,' in Robotics: Science and Systems XIV , vol. 14, 2018. [Online]. Available: http://arxiv.org/abs/1802.09564 - · arXiv
  25. [27][27] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese, 'Neural task programming: Learning to generalize across hierarchical tasks,' in ICRA . IEEE, 2018, pp. 1-8. -
  26. [28][28] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas, 'Playing hard exploration games by watching youtube,' in Advances in Neural Information Processing Systems , 2018, pp. 2935-2945. -
  27. [29][29] A. Hundt, V. Jain, C.-H. Lin, C. Paxton, and G. D. Hager, 'The costar block stacking dataset: Learning with workspace constraints,' Intelligent Robots and Systems (IROS), 2019 IEEE International Conference on , 2019. [Online]. Available: https://arxiv.org/abs/1810.11714 - · arXiv
  28. [30][30] J. Andreas, D. Klein, and S. Levine, 'Modular multitask reinforcement learning with policy sketches,' in ICML'17 Proceedings of the 34th International Conference on Machine Learning - Volume 70 , 2017, pp. 166-175. -
  29. [31][31] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine, 'Learning modular neural network policies for multi-task and multi-robot transfer,' in ICRA . IEEE, 2017, pp. 2169-2176. -
  30. [32][32] D. Amodei, C. Olah, J. Steinhardt, P. F. Christiano, J. Schulman, and D. Man, 'Concrete problems in ai safety,' arXiv preprint arXiv:1606.06565 , 2016. - · arXiv
  31. [33][33] J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg, 'Ai safety gridworlds,' arXiv preprint arXiv:1711.09883 , 2017. - · arXiv
  32. [34][34] T. Everitt, G. Lea, and M. Hutter, 'Agi safety literature review,' in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence , 2018, pp. 5441-5449. -
  33. [35][35] M. Hessel, J. Modayil, H. V. hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, and D. Silver, 'Rainbow: Combining improvements in deep reinforcement learning,' in AAAI , 2018, pp. 3215-3222. -
  34. [36][36] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramr, R. Hadsell, N. de Freitas, and N. Heess, 'Reinforcement and imitation learning for diverse visuomotor skills,' in Robotics: Science and Systems XIV , vol. 14, 2018. -
  35. [37][37] J. Matas, S. James, and A. J. Davison, 'Sim-to-real reinforcement learning for deformable object manipulation,' Conference on Robot Learning , pp. 734-743, 2018. -
  36. [38][38] J. Mahler and K. Goldberg, 'Learning deep policies for robot bin picking by simulating robust grasping sequences,' Conference on Robot Learning , pp. 515-524, 2017. -
  37. [39][39] A. Y. Ng, D. Harada, and S. J. Russell, 'Policy invariance under reward transformations: Theory and application to reward shaping,' in ICML , 1999, pp. 278-287. -
  38. [40][40] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, 'Prioritized experience replay,' in ICLR 2016 : International Conference on Learning Representations 2016 , 2016. -
  39. [41][41] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, 'Scalable deep reinforcement learning for vision-based robotic manipulation,' in Conference on Robot Learning , 2018. -
  40. [43][43] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, 'Hindsight experience replay,' in Advances in Neural Information Processing Systems , 2017, pp. 5048-5058. -
  41. [44][44] C. Paxton, F. Jonathan, A. Hundt, B. Mutlu, and G. D. Hager, 'Evaluating methods for end-user creation of robot task plans,' in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 6086-6092.

Cited by

Sorted by citations

Metadata

Authors
Hundt, A., Killeen, B., Greene, N., Wu, H., Kwon, H., Paxton, C., & Hager, G. D.
Affiliations
  • Johns Hopkins University (US)
  • Nvidia (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
5
Issue
4
Pages
6724-6731
ISSN
2377-3766 (ISSN-L)
DOI
10.48550/arxiv.1909.11730
arXiv
1909.11730
OpenAlex
W3008535267
Cited by
59
References
0
Citations by year
2026: 3 · 2025: 5 · 2024: 8 · 2023: 13 · 2022: 14 · 2021: 12 · 2020: 4
FWCI
4.22
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Office of Naval Research (N00014-17-1-2124)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sim-to-real transfer, block stacking, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review