Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo − Dex, a framework for dexter-ous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-purpose dexterous tasks. Given these demonstrations, we use powerful feature learning combined with non-parametric imi-tation to train dexterous skills. Our experiments on six common dexterous tasks, including in-hand rotation, spinning, and bottle opening, indicate that HOLO-DEX can both collect high-quality demonstration data and train skills in a matter of hours. Finally, we find that our trained skills can exhibit generalization on objects not seen in training. Videos of HOLO − DEX are available on {https://holo-dex.github.io/.}
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 8×
“Hence we select BYOL as our base self-supervised learning method.”
MethodInfluentialEnd to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648 · cited 1×
“To select the learning algorithm for obtaining lowdimensional embeddings (see Section III-A), we experiment with several state-of-the-art self-supervised learning algorithms [63, 66, 67, 68] and find that BYOL [63] provides”
MethodALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Dynamic Few-Shot Visual Learning Without Forgetting
Gidaris, S., & Komodakis, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00459 · Cited by 1,128
A novel type of compliant and underactuated robotic hand for dexterous grasping
Deimel, R., & Brock, O. (2016) · The International Journal of Robotics Research · doi:10.1177/0278364915592961 · Cited by 1,084 · cited 1×
“Due to its high-dimensional action space, learning complex skills with dexterous multi-fingered robot hand has been a longstanding challenge [55, 56, 57, 58].”
BackgroundDota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065 · cited 1×
“in game play [9, 10], text generation [11, 12] or few-shot vision [13, 14].”
Background
Cited by
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2304.13705 · Cited by 673
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System
Qin, Y., Yang, W., Huang, B., Wyk, K., Su, H., Wang, X., Chao, Y.-W., & Fox, D. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.015 · Cited by 109
See to touch: Learning tactile dexterity through visual incentives
Guzey, I., Dai, Y., Evans, B., Chintala, S., & Pinto, L. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611407 · Cited by 26
Self-citation
Metadata
- Authors
- Arunachalam, S. P., Güzey, I., Chintala, S., & Pinto, L.
- Venue
- 2023 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2023-05-29
- Pages
- 5962-5969
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W4383066442
- Cited by
- 45
- References
- 25
- FWCI
- 20.23
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Holo-Dex: Teaching Dexterity with Immersive Mixed Reality. https://app.cochat.ai/papers/10.1109/icra48891.2023.10160547
- The paper
- Arunachalam, S. P., Güzey, I., Chintala, S., & Pinto, L. (2023). Holo-Dex: Teaching Dexterity with Immersive Mixed Reality. 2023 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra48891.2023.10160547
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.