CoChat
Sign Up
Cited by 104Robot Manipulation and LearningReinforcement Learning in RoboticsTeleoperation and Haptic Systems

From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation From Single-Camera Teleoperation

Open access
Qin, Yuzhe, Su, Hao, Wang, Xiaolong·2022·IEEE Robotics and Automation Letters
DOI 10.1109/lra.2022.3196104
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the simulator, which is a manipulator resembling the same structure of the operator's hand. It provides an intuitive interface and avoid unstable human-robot hand retargeting for data collection, leading to large-scale and high quality data. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured) to generate training demonstrations. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks. Importantly, we show our learned policy is significantly more robust when transferring to the real robot.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/lra.2022.3196104

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Robotic Telekinesis systems cost around $18k with one arm and hand.

    “We estimate the cost of these systems to be around $18k with one arm+hand.”

    From Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023) · cites this paper as [48]

    The Introduction (p. 1) says vision-based teleoperation reduces cost, but gives no price. The References (p. 10) list a work titled “Robotic Telekinesis” without discussing its hardware cost. Searches for $18k and the cost of a one-arm-and-hand system found no supporting passage in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 36 · Cited by 104

Research works printed but not linked: 42

  1. [1][1] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. J´ ozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, 'Learning dexterous in-hand manipulation,' arXiv , 2018. -
  2. [2][2] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang, 'Solving rubik's cube with a robot hand,' arXiv , 2019. -
  3. [4][4] V. Kumar and E. Todorov, 'Mujoco haptix: A virtual reality system for hand manipulation,' in International Conference on Humanoid Robots (Humanoids) , 2015. -
  4. [5][5] H. Hedayati, M. Walker, and D. Szafir, 'Improving collocated robot teleoperation with augmented reality,' in International Conference on Human-Robot Interaction , 2018. -
  5. [6][6] Y. Pan, C. Chen, D. Li, Z. Zhao, and J. Hong, 'Augmented realitybased robot teleoperation system using rgb-d imaging and attitude teaching device,' Robotics and Computer-Integrated Manufacturing , vol. 71, p. 102167, 2021. -
  6. [7][7] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, 'Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,' in ICRA , 2018. -
  7. [8][8] A. Handa, K. Van Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox, 'Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system,' in ICRA , 2020. -
  8. [9][9] S. Li, X. Ma, H. Liang, M. G¨ orner, P. Ruppel, B. Fang, F. Sun, and J. Zhang, 'Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network,' in ICRA , 2019. -
  9. [10][10] D. Antotsiou, G. Garcia-Hernando, and T.-K. Kim, 'Task-oriented hand motion retargeting for dexterous manipulation imitation,' in ECCV Workshops , 2018. -
  10. [12][12] V. Kumar, Z. Xu, and E. Todorov, 'Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands,' in ICRA , 2013. -
  11. [14][14] D. Rus, 'In-hand dexterous manipulation of piecewise-smooth 3-d objects,' The International Journal of Robotics Research , 1999. -
  12. [15][15] A. M. Okamura, N. Smaby, and M. R. Cutkosky, 'An overview of dexterous manipulation,' in ICRA , 2000. -
  13. [18][18] D. A. Pomerleau, 'Alvinn: An autonomous land vehicle in a neural network,' in NeurIPS , 1989. -
  14. [19][19] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto, 'Visual imitation made easy,' arXiv , 2020. -
  15. [23][23] J. Fu, K. Luo, and S. Levine, 'Learning robust rewards with adversarial inverse reinforcement learning,' arXiv , 2017. -
  16. [24][24] F. Torabi, G. Warnell, and P. Stone, 'Generative adversarial imitation from observation,' arXiv , 2018. -
  17. [25][25] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas, 'Playing hard exploration games by watching youtube,' in NeurIPS , 2018. -
  18. [28][28] M. Veˇ cer´ ık, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Roth¨ orl, T. Lampe, and M. Riedmiller, 'Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,' arXiv , 2017. -
  19. [29][29] I. Radosavovic, X. Wang, L. Pinto, and J. Malik, 'State-only imitation learning for dexterous manipulation,' IROS , 2021. -
  20. [30][30] K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn, 'Reinforcement learning with videos: Combining offline observations with interaction,' arXiv , 2020. -
  21. [32][32] S. Song, A. Zeng, J. Lee, and T. Funkhouser, 'Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations,' Robotics and Automation Letters , 2020. -
  22. [34][34] Y. Liu, A. Gupta, P. Abbeel, and S. Levine, 'Imitation from observation: Learning to imitate behaviors from raw video via context translation,' in ICRA , 2018. -
  23. [35][35] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, 'Zero-shot visual imitation,' in ICLR , 2018. -
  24. [36][36] P. Sharma, L. Mohan, L. Pinto, and A. Gupta, 'Multiple interactions made easy (mime): Large scale demonstrations data for imitation,' arXiv , 2018. -
  25. [37][37] G. Garcia-Hernando, E. Johns, and T.-K. Kim, 'Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning,' arXiv , 2020. -
  26. [38][38] M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki, 'Graphstructured visual imitation,' in CoRL , 2020. -
  27. [39][39] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg, 'Learning by watching: Physical imitation of manipulation skills from human videos,' arXiv , 2021. -
  28. [40][40] Z. Zhang, 'Microsoft kinect sensor and its effect,' IEEE multimedia , vol. 19, no. 2, pp. 4-10, 2012. -
  29. [41][41] J. Kofman, S. Verma, and X. Wu, 'Robot-manipulator teleoperation by markerless vision-based hand-arm tracking,' International Journal of Optomechatronics , 2007. -
  30. [42][42] G. Du, P. Zhang, J. Mai, and Z. Li, 'Markerless kinect-based hand tracking for robot teleoperation,' International Journal of Advanced Robotic Systems , vol. 9, no. 2, p. 36, 2012. -
  31. [43][43] G.-L. Du, P. Zhang, L.-Y. Yang, and Y.-B. Su, 'Robot teleoperation using a vision-based manipulation method,' in nternational Conference on Audio, Language and Image Processing , 2010. -
  32. [45][45] A. Sivakumar, K. Shaw, and D. Pathak, 'Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube,' arXiv preprint arXiv:2202.10448 , 2022. - · arXiv
  33. [46][46] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto, 'Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation,' arXiv preprint arXiv:2203.13251 , 2022. - · arXiv
  34. [48][48] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, 'Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols,' arXiv , 2015. -
  35. [49][49] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, 'Mediapipe hands: On-device real-time hand tracking,' arXiv preprint arXiv:2006.10214 , 2020. - · arXiv
  36. [50][50] Y. Rong, T. Shiratori, and H. Joo, 'Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration,' arXiv preprint arXiv:2008.08324 , 2020. - · arXiv
  37. [51][51] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black, 'Expressive body capture: 3d hands, face, and body from a single image,' in CVPR , 2019. -
  38. [52][52] S. Kockara, T. Halic, K. Iqbal, C. Bayrak, and R. Rowe, 'Collision detection: A survey,' in International Conference on Systems, Man and Cybernetics , 2007. -
  39. [54][54] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang, 'Dexmv: Imitation learning for dexterous manipulation from human videos,' arXiv preprint arXiv:2108.05877 , 2021. - · arXiv
  40. [55][55] R. M. Murray, Z. Li, and S. S. Sastry, A mathematical introduction to robotic manipulation . CRC press, 2017. -
  41. [56][56] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, 'Trust region policy optimization,' in ICML , 2015. -
  42. [57][57] I.-M. Chen and J. W. Burdick, 'Finding antipodal point grasps on irregularly shaped objects,' IEEE transactions on Robotics and Automation , vol. 9, no. 4, pp. 507-512, 1993. -

Authors
Qin, Y., Su, H., & Wang, X.
Affiliations
University of California San Diego (US)
ORCID
Venue
IEEE Robotics and Automation Letters
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-10
Volume
7
Issue
4
Pages
10873-10881
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
DOI
10.1109/lra.2022.3196104
OpenAlex
W4225003209
Cited by
104
References
36
Citations by year
2026: 14 · 2025: 37 · 2024: 35 · 2023: 15 · 2022: 3
FWCI
9.07
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • NSF (CCF-2112665)
  • Qualcomm and Meta
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
Keywords
imitation learning, dexterous manipulation, policy transfer, human-robot interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review