CoChat
Sign Up
PreprintCited by 42Robot Manipulation and LearningReinforcement Learning in RoboticsDomain Adaptation and Few-Shot Learning

Visual imitation made easy

Open access
Young, Sarah, Gandhi, Dhiraj, Tulsiani, Shubham, Gupta, Abhinav, Abbeel, Pieter, Pinto, Lerrel·2020·arXiv
DOI 10.48550/arxiv.2008.04899arXiv 2008.04899
Open this paper in CoChat — free accountOpen PDF

Abstract

Visual imitation learning provides a framework for learning complex manipulation behaviors by leveraging human demonstrations. However, current interfaces for imitation such as kinesthetic teaching or teleoperation prohibitively restrict our ability to efficiently collect large-scale data in the wild. Obtaining such diverse demonstration data is paramount for the generalization of learned skills to novel scenarios. In this work, we present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. To extract action information from these visual demonstrations, we use off-the-shelf Structure from Motion (SfM) techniques in addition to training a finger detection network. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task. For both tasks, we use standard behavior cloning to learn executable policies from the previously collected offline demonstrations. To improve learning performance, we employ a variety of data augmentations and provide an extensive analysis of its effects. Finally, we demonstrate the utility of our interface by evaluating on real robotic scenarios with previously unseen objects and achieve a 87% success rate on pushing and a 62% success rate on stacking. Robot videos are available at https://dhiraj100892.github.io/Visual-Imitation-Made-Easy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2008.04899

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Handheld tools for demonstration still need a system-specific tool or modifications to the robot's original hardware to align the end-effector's physical interactions with the recorded visual observations.

    “While these approaches simplify tool design, they still require a system-specific tool or modifications to a robot's original hardware to align the end-effector's physical interactions and the recorded visual observations.”

    From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [12]

    The Introduction says direct transfer depends on the robot having the same tool setup as the demonstration tool. Appendix B describes attaching that tool to the robot end-effector and modifying its camera mount, but it does not establish that the tool is system-specific or that the robot’s original hardware must be modified; the Conclusion calls the tool usable with virtually any robot. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 34

  1. [1][1] J. Piaget. Play, dreams and imitation in childhood, volume 25. Routledge, 2013.
  2. [2][2] A. N. Meltzoff and M. K. Moore. Imitation of facial and manual gestures by human neonates. Science, 198(4312):75–78, 1977.
  3. [3][3] A. N. Meltzoff and M. K. Moore. Newborn infants imitate adult facial gestures. Child development, pages 702–709, 1983.
  4. [4][4] M. Tomasello, S. Savage-Rumbaugh, and A. C. Kruger. Imitative learning of actions on objects by children, chimpanzees, and enculturated chimpanzees. Child development, 64(6):1688–1705, 1993.
  5. [5][5] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
  6. [6][6] B. C. Stadie, P. Abbeel, and I. Sutskever. Third-person imitation learning. ICLR, 2017.
  7. [7][7] Y. Zhu, Z. Wang, J. Merel, A. A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, abs/1802.09564, 2018. URL http://arxiv.org/abs/1802.09564. · arXiv
  8. [8][8] A. Gupta, A. Murali, D. P. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. In Advances in Neural Information Processing Systems, pages 9094–9104, 2018.
  9. [9][9] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters (RA-L), 2020.
  10. [10][10] P. Sharma, L. Mohan, L. Pinto, and A. Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. CoRL, 2018.
  11. [11][11] P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv:1612.06699, 2016. · arXiv
  12. [13][13] F. Ruggiero, V. Lippiello, and B. Siciliano. Nonprehensile dynamic manipulation: A survey. IEEE Robotics and Automation Letters, 3(3):1711–1718, 2018.
  13. [14][14] Y. Zhang, B. K. Chen, X. Liu, and Y. Sun. Autonomous robotic pick-and-place of microobjects. IEEE Transactions on Robotics, 26(1):200–207, 2009.
  14. [15][15] F. Furrer, M. Wermelinger, H. Yoshida, F. Gramazio, M. Kohler, R. Siegwart, and M. Hutter. Autonomous robotic stone stacking with online next best object target pose planning. In ICRA, 2017.
  15. [18][18] P. K. Pook and D. H. Ballard. Recognizing teleoperated manipulations. In [1993] Proceedings IEEE International Conference on Robotics and Automation, pages 578–585. IEEE, 1993.
  16. [19][19] J. D. Sweeney and R. Grupen. A model of shared grasp affordances from demonstration. In 2007 7th IEEE-RAS International Conference on Humanoid Robots, pages 27–35. IEEE, 2007.
  17. [22][22] S. Ross, N. Melik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert. Learning monocular reactive uav control in cluttered natural environments. In ICRA, 2013. 9 http://arxiv.org/abs/1802.09564 · arXiv
  18. [23][23] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016. · arXiv
  19. [24][24] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dex- pilot: Vision based teleoperation of dexterous robotic hand-arm system. arXiv preprint arXiv:1910.03135, 2019. · arXiv
  20. [25][25] B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz. Keyframe-based learning from demonstration. International Journal of Social Robotics, 4(4):343–355, 2012.
  21. [26][26] K. Muelling, A. Boularias, B. Mohler, B. Schölkopf, and J. Peters. Learning strategies in table tennis using inverse reinforcement learning. Biological cybernetics, 108(5):603–619, 2014.
  22. [28][28] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635, 2011.
  23. [29][29] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
  24. [31][31] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
  25. [32][32] D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In ICML, 2019.
  26. [33][33] B. Zoph, E. D. Cubuk, G. Ghiasi, T.-Y. Lin, J. Shlens, and Q. V. Le. Learning data augmentation strategies for object detection. arXiv preprint arXiv:1906.11172, 2019. · arXiv
  27. [34][34] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
  28. [35][35] Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848, 2019. · arXiv
  29. [36][36] F. Sadeghi and S. Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201, 2016. · arXiv
  30. [39][39] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004.14990, 2020. · arXiv
  31. [40][40] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020. · arXiv
  32. [45][45] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision (ECCV), 2016. 10 https://www.amazon.com/RMS-Grabber-Reacher-Rotating-Gripper/dp/B00THEDNL8/ref=sr_1_16?dchild=1&keywords=grabber+tool&qid=1595965487&sr=8-16 https://www.amazon.com/RMS-Grabber-Reacher-Rotating-Gripper/dp/B00THEDNL8/ref=sr_1_16?dchild=1&keywords=grabber+tool&qid=1595965487&sr=8-16 https://gopro.com/en/us/shop/cameras/hero7-silver/CHDHC-601-master.html https://gopro.com/en/us/shop/cameras/hero7-silver/CHDHC-601-master.html https://www.xarm.cc/products/xarm-7-2020
  33. [46][46] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305–313, 1989.
  34. [47][47] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2019.

Cited by

Sorted by citations

Authors
Young, S., Gandhi, D., Tulsiani, S., Gupta, A., Abbeel, P., & Pinto, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
1992-2005
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2008.04899
arXiv
2008.04899
OpenAlex
W3048833305
Cited by
42
References
4
Citations by year
2025: 3 · 2024: 10 · 2023: 11 · 2022: 6 · 2021: 11 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Conference on Robot Learning
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
visual imitation learning, structure from motion, finger detection, behavior cloning, data augmentation, robotic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review