CoChat
Sign Up
PreprintCited by 618Reinforcement Learning in RoboticsRobot Manipulation and LearningTeleoperation and Haptic Systems

Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

Open access
Zhang, Tianhao, McCarthy, Zoe, Jow, Owen, Lee, Dennis, Chen, Xi, Goldberg, Ken, Abbeel, Pieter·2017·arXiv
DOI 10.48550/arxiv.1710.04615arXiv 1710.04615
Open this paper in CoChat — free accountOpen PDF

Abstract

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1710.04615

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The tasks MimicGen was demonstrated on have pose variation well beyond typical prior works using BC from human demos.

    “The tasks also have pose variation well beyond typical prior works using BC from human demos [1,3-7,30,33,55,56].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [1]

    The Appendix (Task Specification; page unavailable) describes varying object positions and orientations across several tasks, including a hammer placed within a 15 × 15 cm area at varying angles. The paper does not establish that this pose variation is well beyond that of typical prior work using behavioral cloning from human demonstrations. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • A large number of robot learning works and benchmarks use quasi-static tasks, making the system broadly applicable.

    “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3-7,14,18,19,22,30,33,51, 55, 56, 64-66], making the system broadly applicable.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [1]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. S. Schaal, "Is imitation learning the route to humanoid robots?" Trends in cognitive sciences, vol. 3, no. 6, pp. 233-242, 1999.
  2. S. Calinon, Robot programming by demonstration. EPFL Press, 2009.
  3. B. D. Argall, S. Chernova, M. Veloso, and B. Browning, "A survey of robot learning from demonstration," Robotics and autonomous systems, vol. 57, no. 5, pp. 469-483, 2009.
  4. D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," in Advances in Neural Information Processing Systems, 1989, pp. 305-313.
  5. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al., "End to end learning for self-driving cars," arXiv preprint arXiv:1604.07316, 2016. · arXiv
  6. A. Giusti, J. Guzzi, D. C. Cires ¸an, F.-L. He, J. P. Rodríguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro, et al., "A machine learning approach to visual perception of forest trails for mobile robots," IEEE Robotics and Automation Letters, vol. 1, no. 2, pp. 661-667, 2016.
  7. P. Abbeel, A. Coates, and A. Y. Ng, "Autonomous helicopter aerobatics through apprenticeship learning," The International Journal of Robotics Research, 2010.
  8. S. Calinon, F. D'halluin, E. L. Sauser, D. G. Caldwell, and A. G. Billard, "Learning and reproduction of gestures by imitation," IEEE Robotics & Automation Magazine, vol. 17, no. 2, pp. 44-54, 2010.
  9. B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz, "Keyframe-based learning from demonstration," International Journal of Social Robotics, vol. 4, no. 4, pp. 343-355, 2012.
  10. A. Dragan, K. C. Lee, and S. Srinivasa, "Teleoperation with intelligent and customizable interfaces," Journal of Human-Robot Interaction, vol. 1, no. 3, 2013.
  11. A. E. Bryson, Applied optimal control: optimization, estimation and control. CRC Press, 1975.
  12. M. Posa, C. Cantu, and R. Tedrake, "A direct method for trajectory optimization of rigid bodies through contact," The International Journal of Robotics Research, vol. 33, no. 1, pp. 69-81, 2014.
  13. S. Levine and P. Abbeel, "Learning neural network policies with guided policy search under unknown dynamics," in Advances in Neural Information Processing Systems, 2014, pp. 1071-1079.
  14. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, no. 7, pp. 1180-1190, 2008.
  15. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015.
  16. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, "Trust region policy optimization," in Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 2015, pp. 1889-1897.
  17. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforcement learning," arXiv preprint arXiv:1509.02971, 2015. · arXiv
  18. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," Journal of Machine Learning Research, vol. 17, no. 39, pp. 1-40, 2016.
  19. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning," in International Conference on Machine Learning, 2016, pp. 1928-1937.
  20. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Prox- imal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017. · arXiv
  21. M. Talamini, K. Campbell, and C. Stanfield, "Robotic gastrointestinal surgery: early experience and system description," Journal of laparoen- doscopic & advanced surgical techniques, vol. 12, no. 4, pp. 225-232, 2002.
  22. S. Ross, G. J. Gordon, and D. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning." in AISTATS, vol. 1, no. 2, 2011, p. 6.
  23. A. Y. Ng, S. J. Russell, et al., "Algorithms for inverse reinforcement learning." in Icml, 2000, pp. 663-670.
  24. P. Abbeel and A. Ng, "Apprenticeship learning via inverse reinforcement learning," in International Conference on Machine Learning (ICML), 2004.
  25. B. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforcement learning," in AAAI Conference on Artificial Intelligence, 2008.
  26. S. Levine, Z. Popovic, and V. Koltun, "Nonlinear inverse reinforcement learning with gaussian processes," in Advances in Neural Information Processing Systems (NIPS), 2011.
  27. C. Finn, S. Levine, and P. Abbeel, "Guided cost learning: Deep inverse optimal control via policy optimization," in Proceedings of the 33rd International Conference on Machine Learning, vol. 48, 2016.
  28. J. Ho and S. Ermon, "Generative adversarial imitation learning," in Advances in Neural Information Processing Systems, 2016, pp. 4565- 4573.
  29. A. Billard, Y. Epars, S. Calinon, S. Schaal, and G. Cheng, "Discovering optimal imitation strategies," Robotics and autonomous systems, vol. 47, no. 2, pp. 69-77, 2004.
  30. S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert, "Learning movement primitives," Robotics Research, pp. 561-572, 2005.
  31. P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, "Learning and gener- alization of motor skills by learning from demonstration," in Robotics and Automation, 2009. ICRA'09. IEEE International Conference on. IEEE, 2009, pp. 763-768.
  32. N. Ratliff, J. A. Bagnell, and S. S. Srinivasa, "Imitation learning for locomotion and manipulation," in Humanoid Robots, 2007 7th IEEE-RAS International Conference on. IEEE, 2007, pp. 392-397.
  33. T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, et al., "Learning from demonstrations for real world reinforcement learning," arXiv preprint arXiv:1704.03732, 2017. · arXiv
  34. S. Ross, N. Melik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert, "Learning monocular reactive uav control in cluttered natural environments," in Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 2013, pp. 1765-1772.
  35. R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine, "Vision- based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration," arXiv preprint arXiv:1707.02920, 2017. · arXiv
  36. B. C. Stadie, P. Abbeel, and I. Sutskever, "Third-person imitation learning," arXiv preprint arXiv:1703.01703, 2017. · arXiv
  37. Y. Liu, A. Gupta, P. Abbeel, and S. Levine, "Imitation from observation: Learning to imitate behaviors from raw video via context translation," arXiv preprint arXiv:1707.03374, 2017. · arXiv
  38. C. Stanton, A. Bogdanovych, and E. Ratanasena, "Teleoperation of a humanoid robot using full-body motion capture, example movements, and machine learning," in Proc. Australasian Conference on Robotics and Automation, 2012.
  39. L. Fritsche, F. Unverzag, J. Peters, and R. Calandra, "First-person tele- operation of a humanoid robot," in Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International Conference on. IEEE, 2015, pp. 997-1002.
  40. J. I. Lipton, A. J. Fay, and D. Rus, "Baxter's homunculus: Virtual reality spaces for teleoperation in manufacturing," IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 179-186, 2018.
  41. V. Kumar and E. Todorov, "Mujoco haptix: A virtual reality system for hand manipulation," in Humanoid Robots (Humanoids), 2015 IEEE- RAS 15th International Conference on. IEEE, 2015, pp. 657-663.
  42. E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris, and S. Tellex, "Communicating robot arm motion intent through mixed reality head-mounted displays," arXiv preprint arXiv:1708.03655, 2017. · arXiv
  43. X. Yan, M. Khansari, Y. Bai, J. Hsu, A. Pathak, A. Gupta, J. Davidson, and H. Lee, "Learning grasping interaction with geometry-aware 3d representations," arXiv preprint arXiv:1708.07303, 2017. · arXiv
  44. V. Kumar, A. Gupta, E. Todorov, and S. Levine, "Learning dexterous manipulation policies from experience and imitation," in ICRA, 2016.
  45. A. Y. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, and E. Liang, "Autonomous inverted helicopter flight via reinforcement learning," in Experimental Robotics IX. Springer Berlin Heidelberg, 2006, pp. 363-372.
  46. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients," Neural networks, vol. 21, no. 4, pp. 682-697, 2008.
  47. R. Tedrake, T. W. Zhang, and H. S. Seung, "Stochastic policy gradient reinforcement learning on a simple 3d biped," in Intelligent Robots and Systems, 2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ International Conference on, vol. 3. IEEE, 2004, pp. 2849-2854.
  48. A. Y. Ng and S. J. Russell, "Algorithms for inverse reinforcement learning." in Icml, 2000, pp. 663-670.
  49. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Sil- ver, and K. Kavukcuoglu, "Reinforcement learning with unsupervised auxiliary tasks," arXiv preprint arXiv:1611.05397, 2016. · arXiv
  50. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1-9.
  51. D. Kingma and J. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015.

Cited by

Sorted by citations

Authors
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., & Abbeel, P.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
5628-5635
ISSN
1050-4729 (ISSN-L)
DOI
10.48550/arxiv.1710.04615
arXiv
1710.04615
OpenAlex
W2963669336
Cited by
618
References
1
Citations by year
2026: 29 · 2025: 90 · 2024: 85 · 2023: 68 · 2022: 87 · 2021: 87 · 2020: 83 · 2019: 69 · 2018: 19 · 2016: 1
FWCI
25.67
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Office of Naval Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
Keywords
imitation learning, deep reinforcement learning, virtual reality teleoperation, hand tracking, visuomotor skills, robotic manipulation, skill acquisition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review