CoChat
Sign Up
PreprintCited by 48Robot Manipulation and LearningTeleoperation and Haptic SystemsReinforcement Learning in Robotics

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

Open access
Arunachalam, Sridhar Pandian, Güzey, Irmak, Chintala, Soumith, Pinto, Lerrel·2022·arXiv
DOI 10.48550/arxiv.2210.06463arXiv 2210.06463
Open this paper in CoChat — free accountOpen PDF

Abstract

A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo-Dex, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-purpose dexterous tasks. Given these demonstrations, we use powerful feature learning combined with non-parametric imitation to train dexterous skills. Our experiments on six common dexterous tasks, including in-hand rotation, spinning, and bottle opening, indicate that Holo-Dex can both collect high-quality demonstration data and train skills in a matter of hours. Finally, we find that our trained skills can exhibit generalization on objects not seen in training. Videos of Holo-Dex are available at https://holo-dex.github.io.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.06463

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Robotic Telekinesis systems cost around $18k with one arm and hand.

    “We estimate the cost of these systems to be around $18k with one arm+hand.”

    From Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023) · cites this paper as [6]

    The Dexterous Teleoperation Frameworks section says VR hand tracking is cheaper than a CyberGlove, and Robot Hand Control describes the Allegro hand, but neither gives a price for a one-arm-and-hand system. Searches for $18k, $18,000, and arm-and-hand costs found no cost estimate in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 56

  1. C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, "Iterative residual policy: for goal-conditioned dy- namic manipulation of deformable objects," arXiv preprint arXiv:2203.00663, 2022. · arXiv
  2. A. E. Tekden, A. Erdem, E. Erdem, M. Imre, M. Y. Seker, and E. Ugur, "Belief regulated dual propagation nets for learning action effects on groups of articulated objects," in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 10 556-10 562.
  3. S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis, "Rloc: Terrain-aware legged locomotion using re- inforcement learning and optimal control," IEEE Transactions on Robotics, 2022.
  4. Y. Ma, F. Farshidian, T. Miki, J. Lee, and M. Hutter, "Com- bining learning-based locomotion policy with model-based manipulation for legged mobile manipulators," IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2377-2384, 2022.
  5. L. Smith, I. Kostrikov, and S. Levine, "A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning," arXiv preprint arXiv:2208.07860, 2022. · arXiv
  6. T. Zhang, G. Kahn, S. Levine, and P. Abbeel, "Learning deep control policies for autonomous aerial vehicles with mpc- guided policy search," in 2016 IEEE international conference on robotics and automation (ICRA). IEEE, 2016, pp. 528- 535.
  7. D. Gandhi, L. Pinto, and A. Gupta, "Learning to fly by crashing," in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 3948-3955.
  8. J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, "Control of a quadrotor with reinforcement learning," IEEE Robotics and Automation Letters, vol. 2, no. 4, pp. 2096-2103, 2017.
  9. C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al., "Dota 2 with large scale deep reinforcement learning," arXiv preprint arXiv:1912.06680, 2019. · arXiv
  10. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneer- shelvam, M. Lanctot, et al., "Mastering the game of go with deep neural networks and tree search," nature, vol. 529, no. 7587, pp. 484-489, 2016.
  11. J. Guo, S. Lu, H. Cai, W. Zhang, Y. Yu, and J. Wang, "Long text generation via adversarial training with leaked informa- tion," in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.
  12. M. Huang, F. Li, W. Zou, and W. Zhang, "Sarg: A novel semi autoregressive generator for multi-turn incomplete utterance restoration," in Proceedings of the AAAI Conference on Arti- ficial Intelligence, vol. 35, no. 14, 2021, pp. 13 055-13 063.
  13. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020. · arXiv
  14. S. Gidaris and N. Komodakis, "Dynamic few-shot visual learning without forgetting," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4367-4375.
  15. L. Pinto and A. Gupta, "Supersizing self-supervision: Learn- ing to grasp from 50k tries and 700 robot hours," ICRA, 2016.
  16. A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine, "End-to-end robotic reinforcement learning without reward engineering," arXiv preprint arXiv:1904.07854, 2019. · arXiv
  17. M. Vecerik, O. Sushkov, D. Barker, T. Rothörl, T. Hester, and J. Scholz, "A practical approach to insertion with variable socket position using deep reinforcement learning," in 2019 international conference on robotics and automation (ICRA). IEEE, 2019, pp. 754-760.
  18. G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Padu- raru, S. Gowal, and T. Hester, "An empirical investigation of the challenges of real-world reinforcement learning," arXiv preprint arXiv:2003.11881, 2020. · arXiv
  19. OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, "Learning dexterous in-hand manipulation," arXiv, 2018.
  20. OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang, "Solving rubik's cube with a robot hand," arXiv, 2019.
  21. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, "Domain randomization for transferring deep neu- ral networks from simulation to the real world," in IROS, 2017.
  22. N. Jakobi, P. Husbands, and I. Harvey, "Noise and the reality gap: The use of simulation in evolutionary robotics," in Advances in Artificial Life, F. Morán, A. Moreno, J. J. Merelo, and P. Chacón, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 1995, pp. 704-720.
  23. D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," in NeurIPS, 1989, pp. 305-313.
  24. P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforcement learning," in ICML, 2004, p. 1.
  25. A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox, "Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm sys- tem," in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 9164-9170.
  26. S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto, "Dexterous imitation made easy: A learning-based frame- work for efficient dexterous manipulation," arXiv preprint arXiv:2203.13251, 2022. · arXiv
  27. M. Caeiro-Rodríguez, I. Otero-González, F. A. Mikic-Fonte, and M. Llamas-Nistal, "A systematic review of commercial smart gloves: Current status and applications," Sensors, 2021.
  28. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforcement learning," arXiv preprint, 2015.
  29. D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, "Mastering visual continuous control: Improved data-augmented rein- forcement learning," arXiv preprint arXiv:2107.09645, 2021. · arXiv
  30. F. Sadeghi and S. Levine, "Cad2rl: Real single-image flight without a single real image," arXiv preprint arXiv:1611.04201, 2016. · arXiv
  31. Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel, "Learn- ing to manipulate deformable objects without demonstrations," arXiv preprint, 2019.
  32. K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, "Rl-cyclegan: Reinforcement learning aware simulation-to-real," in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2020, pp. 11 157-11 166.
  33. Y. Qin, H. Su, and X. Wang, "From one hand to multiple hands: Imitation learning for dexterous manip- ulation from single-camera teleoperation," arXiv preprint arXiv:2204.12490, 2022. · arXiv
  34. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al., "End to end learning for self-driving cars," arXiv:1604.07316, 2016. · arXiv
  35. S. Schaal and C. Atkeson, "Robot juggling: implementation of memory-based learning," IEEE CSM, 1994.
  36. S. Haldar, V. Mathur, D. Yarats, and L. Pinto, "Watch and match: Supercharging imitation with regularized optimal transport," arXiv preprint arXiv:2206.15469, 2022. · arXiv
  37. S. Li, J. Jiang, P. Ruppel, H. Liang, X. Ma, N. Hendrich, F. Sun, and J. Zhang, "A mobile robot hand-arm teleoperation system by vision and imu," in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 10 900-10 906.
  38. S. Li, X. Ma, H. Liang, M. Görner, P. Ruppel, B. Fang, F. Sun, and J. Zhang, "Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network," in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 416-422.
  39. T. Hentschel and J. A. Neuhöfer, "Steady hands -an evaluation on the use of hand tracking in virtual reality training in nursing," in Proceedings of the 21st Congress of the Interna- tional Ergonomics Association (IEA 2021), N. L. Black, W. P. Neumann, and I. Noy, Eds. Cham: Springer International Publishing, 2022, pp. 643-649.
  40. M. Salvato, N. Heravi, A. M. Okamura, and J. Bohg, "Predict- ing hand-object interaction for improved haptic feedback in mixed reality," IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3851-3857, 2022.
  41. Z. Gharaybeh, H. Chizeck, and A. Stewart, "Telerobotic con- trol in virtual reality," in OCEANS 2019 MTS/IEEE SEATTLE, 2019, pp. 1-8.
  42. T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, "Deep imitation learning for complex manipu- lation tasks from virtual reality teleoperation," in ICRA, 2018.
  43. I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell, "Real-world robot learning with masked visual pre-training," 2022. [Online]. Available: https://arxiv.org/abs/ 2210.03109
  44. V. Kumar, Y. Tassa, T. Erez, and E. Todorov, "Real-time behaviour synthesis for dynamic hand-manipulation," in 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014, pp. 6808-6815.
  45. I. Mordatch, Z. Popović, and E. Todorov, "Contact-invariant optimization for hand manipulation," in Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer An- imation, ser. SCA '12. Goslar, DEU: Eurographics Associ- ation, 2012, p. 137-144.
  46. R. Deimel and O. Brock, "A novel type of compliant and underactuated robotic hand for dexterous grasping," The In- ternational Journal of Robotics Research, vol. 35, no. 1-3, pp. 161-185, 2016.
  47. J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg, "Learning ambidextrous robot grasping policies," Science Robotics, vol. 4, no. 26, p. eaau4984, 2019.
  48. V. Kumar, A. Gupta, E. Todorov, and S. Levine, "Learning dexterous manipulation policies from experience and imita- tion," arXiv, 2016.
  49. A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar, "Deep dynamics models for learning dexterous manipulation," arXiv, 2019.
  50. I. Radosavovic, X. Wang, L. Pinto, and J. Malik, "State- only imitation learning for dexterous manipulation," in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 7865-7871.
  51. L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales, "Self- supervised representation learning: Introduction, advances, and challenges," IEEE Signal Processing Magazine, vol. 39, no. 3, pp. 42-62, 2022.
  52. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al., "Bootstrap your own latent-a new approach to self-supervised learning," NeurIPS, 2020.
  53. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," arXiv preprint, 2020.
  54. X. Chen, S. Xie, and K. He, "An empirical study of training self-supervised vision transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9640-9649.
  55. A. Bardes, J. Ponce, and Y. LeCun, "Vicreg: Variance- invariance-covariance regularization for self-supervised learn- ing," arXiv preprint arXiv:2105.04906, 2021. · arXiv
  56. S. Young, J. Pari, P. Abbeel, and L. Pinto, "Playful interactions for representation learning," arXiv preprint arXiv:2107.09046, 2021. · arXiv

Cited by

Sorted by citations

Authors
Arunachalam, S. P., Güzey, I., Chintala, S., & Pinto, L.
Affiliations
New York University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
5962-5969
ISSN
1050-4729 (ISSN-L)
DOI
10.48550/arxiv.2210.06463
arXiv
2210.06463
OpenAlex
W4383066442
Cited by
48
References
0
Citations by year
2026: 7 · 2025: 23 · 2024: 12 · 2023: 6
FWCI
20.55
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Office of Naval Research
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
dexterous manipulation, robot teaching, teleoperation, hand pose estimation, virtual reality headsets, imitation learning, skill generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review