CoChat
Sign Up
Cited by 109Teleoperation and Haptic SystemsRobot Manipulation and LearningReinforcement Learning in Robotics

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

Open access
Qin, Yuzhe, Yang, Wei, Huang, Binghao, Wyk, Karl, Su, Hao, Wang, Xiaolong, Chao, Yu-Wei, Fox, Dieter·2023·Robotics: Science and Systems XIX
DOI 10.15607/rss.2023.xix.015
Open this paper in CoChat — free account

Abstract

Real World* Yuzhe Qin was an intern at NVIDIA during the project.experiments, AnyTeleop can outperform a previous system that was designed for a specific robot hardware with a higher success rate, using the same robot.For teleoperation in simulation, AnyTeleop leads to better imitation learning performance, compared with a previous system that is particularly designed for that simulator.Robot 1 Controlled by Operator #1 On Computer #1 Robot 2 Controlled by Operator #2 On Computer #2

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.15607/rss.2023.xix.015

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In the retargeting comparison, Vector achieved the best raw fidelity at 10.6 mm but had 29.9% joint-limit violations and 3.46 rad/s jitter.

    “Vector achieves the best raw fidelity (10.6 mm) but incurs 29.9% joint-limit violations and 3.46 rad/s jitter.”

    From Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning (2026) · cites this paper as Qin et al., 2023

    The Hand Pose Retargeting section describes an optimization with joint-limit constraints and a temporal-smoothness penalty, but does not report a comparison involving Vector. Searches for Vector and the figures 10.6 mm, 29.9%, and 3.46 rad/s found no passage in our copy supporting the attributed results; Table II instead reports module runtimes. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 54

  1. Insaf Ajili, Malik Mallem, and Jean-Yves Didier. Ges- ture recognition for humanoid robot teleoperation. In 2017 26Th IEEE international symposium on robot and human interactive communication (RO-MAN), pages 1115-1120. IEEE, 2017.
  2. Dafni Antotsiou, Guillermo Garcia-Hernando, and Tae- Kyun Kim. Task-oriented hand motion retargeting for dexterous manipulation imitation. In ECCV Workshops, 2018.
  3. Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, and Lerrel Pinto. Holo-dex: Teaching dex- terity with immersive mixed reality. arXiv preprint arXiv:2210.06463, 2022. · arXiv
  4. Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, and Lerrel Pinto. Dexterous imitation made easy: A learning-based framework for efficient dexter- ous manipulation. arXiv preprint arXiv:2203.13251, 2022. · arXiv
  5. Sean Chen, Jensen Gao, Siddharth Reddy, Glen Berseth, Anca D Dragan, and Sergey Levine. Asha: As- sistive teleoperation via human-in-the-loop reinforce- ment learning. In 2022 International Conference on Robotics and Automation (ICRA), pages 7505-7512. IEEE, 2022.
  6. Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, and Nathan Ratliff. Rmp flow: A computational graph for au- tomatic motion policy generation. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13, pages 441-457. Springer, 2020.
  7. Brian Danchilla and Brian Danchilla. Three. js frame- work. Beginning WebGL for HTML5, pages 173-203, 2012.
  8. Guanglong Du, Ping Zhang, Jianhua Mai, and Zel- ing Li. Markerless kinect-based hand tracking for robot teleoperation. International Journal of Advanced Robotic Systems, 9(2):36, 2012.
  9. Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Fre- itas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interac- tive multi-modal physical simulation. arXiv preprint arXiv:2007.04954, 2020. · arXiv
  10. Zaid Gharaybeh, Howard Chizeck, and Andrew Stew- art. Telerobotic control in virtual reality. IEEE, 2019.
  11. Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. Dexpilot: Vision-based tele- operation of dexterous robotic hand-arm system. In ICRA, 2020.
  12. Hooman Hedayati, Michael Walker, and Daniel Szafir. Improving collocated robot teleoperation with aug- mented reality. In International Conference on Human- Robot Interaction, 2018.
  13. Zebin Huang, Ziwei Wang, Weibang Bai, Yanpei Huang, Lichao Sun, Bo Xiao, and Eric M Yeatman. A novel training and collaboration integrated framework for human-agent teleoperation. Sensors, 21(24):8341, 2021.
  14. Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kap- pler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
  15. Bachir El Khadir, Jake Varley, and Vikas Sindhwani. Teleoperator imitation with continuous-time safety. arXiv preprint arXiv:1905.09499, 2019. · arXiv
  16. Jonathan Kofman, Xianghai Wu, Timothy J Luu, and Siddharth Verma. Teleoperation of a robot manip- ulator using a vision-based human-robot interface. IEEE transactions on industrial electronics, 52(5): 1206-1219, 2005.
  17. Jonathan Kofman, Siddharth Verma, and Xianghai Wu. Robot-manipulator teleoperation by markerless vision- based hand-arm tracking. International Journal of Optomechatronics, 2007.
  18. Vikash Kumar and Emanuel Todorov. Mujoco haptix: A virtual reality system for hand manipulation. In International Conference on Humanoid Robots (Hu- manoids), 2015.
  19. Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Görner, Philipp Ruppel, Bin Fang, Fuchun Sun, and Jianwei Zhang. Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In ICRA, 2019.
  20. Shuang Li, Jiaxi Jiang, Philipp Ruppel, Hongzhuo Liang, Xiaojian Ma, Norman Hendrich, Fuchun Sun, and Jianwei Zhang. A mobile robot hand-arm teleop- eration system by vision and IMU. In IROS, 2020.
  21. Shuang Li, Norman Hendrich, Hongzhuo Liang, Philipp Ruppel, Changshui Zhang, and Jianwei Zhang. A dexterous hand-arm teleoperation system based on hand pose estimation and active vision. IEEE Trans- actions on Cybernetics, 2022.
  22. Jacky Liang, Ankur Handa, Karl Van Wyk, Viktor Makoviychuk, Oliver Kroemer, and Dieter Fox. In- hand object pose tracking via contact feedback and gpu-accelerated robotic simulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6203-6209. IEEE, 2020.
  23. Jeffrey I Lipton, Aidan J Fay, and Daniela Rus. Bax- ter's homunculus: Virtual reality spaces for teleopera- tion in manufacturing. IEEE Robotics and Automation Letters, 3(1):179-186, 2017.
  24. Hangxin Liu, Xu Xie, Matt Millar, Mark Edmonds, Feng Gao, Yixin Zhu, Veronica J Santos, Brandon Rothrock, and Song-Chun Zhu. A glove-based system for studying hand-object manipulation via joint pose and force sensing. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6617-6624. IEEE, 2017.
  25. Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In CVPR, 2021.
  26. Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021. · arXiv
  27. Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pages 879-893. PMLR, 2018.
  28. Cassie Meeker, Maximilian Haas-Heger, and Matei Ciocarlie. A continuous teleoperation subspace with empirical and algorithmic mapping algorithms for nonanthropomorphic hands. IEEE Transactions on Automation Science and Engineering, 19(1):373-386, 2020.
  29. C Mizera, T Delrieu, V Weistroffer, C Andriot, A De- catoire, and J-P Gazeau. Evaluation of hand-tracking systems in teleoperation and virtual dexterous manipu- lation. IEEE Sensors Journal, 20(3):1642-1655, 2019.
  30. Katharina Muelling, Arun Venkatraman, Jean- Sebastien Valois, John Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B Schwartz, Jennifer L Collinger, and J Andrew Bagnell. Autonomy infused teleoperation with application to bci manipulation. arXiv preprint arXiv:1503.05451, 2015. · arXiv
  31. Günter Niemeyer, Carsten Preusche, Stefano Stramigi- oli, and Dongjun Lee. Telerobotics. Springer handbook of robotics, pages 1085-1108, 2016.
  32. Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
  33. Polina Ponomareva, Daria Trinitatova, Aleksey Fe- doseev, Ivan Kalinov, and Dzmitry Tsetserukou. Gras- plook: a vr-based telemanipulation system with r-cnn- driven augmentation of virtual environment. In 2021 20th International Conference on Advanced Robotics (ICAR), pages 166-171. IEEE, 2021.
  34. Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dex- terous manipulation from single-camera teleoperation. RA-L, 7(4):10873-10881, 2022.
  35. Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. arXiv preprint arXiv:2210.03109, 2022. · arXiv
  36. Daniel Rakita, Bilge Mutlu, and Michael Gleicher. A motion retargeting method for effective mimicry-based teleoperation of robot arms. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, pages 361-370, 2017.
  37. Nathan D Ratliff, Jan Issac, Daniel Kappler, Stan Birchfield, and Dieter Fox. Riemannian motion poli- cies. arXiv preprint arXiv:1801.02854, 2018. · arXiv
  38. Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008.08324, 2020. · arXiv
  39. Eric Rosen, David Whitney, Michael Fishman, Daniel Ullman, and Stefanie Tellex. Mixed reality as a bidirectional communication interface for human-robot interaction. In 2020 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 11431-11438. IEEE, 2020.
  40. M Salvato, Negin Heravi, Allison M Okamura, and Jeannette Bohg. Predicting hand-object interaction for improved haptic feedback in mixed reality. IEEE Robotics and Automation Letters, 7(2):3851-3857, 2022.
  41. Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. arXiv preprint arXiv:2212.04498, 2022. · arXiv
  42. Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: learning a robotic hand im- itator by watching humans on youtube. arXiv preprint arXiv:2202.10448, 2022. · arXiv
  43. Hyoung Il Son, Antonio Franchi, Lewis L Chuang, Junsuk Kim, Heinrich H Bulthoff, and Paolo Robuffo Giordano. Human-centered design and evaluation of haptic cueing for teleoperation of multiple mobile robots. IEEE transactions on cybernetics, 43(2):597- 609, 2013.
  44. Balakumar Sundaralingam, Siva Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexan- der Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, and Dieter Fox. CuRobo: Parallelized collision-free robot motion generation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), June 2023.
  45. Christian Theobalt, Irene Albrecht, Jörg Haber, Marcus Magnor, and Hans-Peter Seidel. Pitching a baseball: tracking high-speed motion with multi-exposure im- ages. In ACM SIGGRAPH 2004 Papers, pages 540- 547. 2004.
  46. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
  47. Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Learning multi-arm manipulation through collaborative teleoperation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9212-9219. IEEE, 2021.
  48. Robert Y Wang and Jovan Popović. Real-time hand- tracking with a color glove. ACM transactions on graphics (TOG), 28(3):1-8, 2009.
  49. Dong Wei, Bidan Huang, and Qiang Li. Multi-view merging for robot teleoperation with virtual reality. IEEE Robotics and Automation Letters, 6(4):8537- 8544, 2021.
  50. Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. Mediapipe hands: On- device real-time hand tracking. arXiv preprint arXiv:2006.10214, 2020. · arXiv
  51. Haodong Zhang, Weijie Li, Yuwei Liang, Zexi Chen, Yuxiang Cui, Yue Wang, and Rong Xiong. Human- robot motion retargeting via neural latent optimization. CoRR, 2021.
  52. Heng Zhang, Zeming Zhao, Yang Yu, Kai Gui, Xinjun Sheng, and Xiangyang Zhu. A feasibility study on an intuitive teleoperation system combining imu with semg sensors. In Intelligent Robotics and Applications: 11th International Conference, ICIRA 2018, Newcastle, NSW, Australia, August 9-11, 2018, Proceedings, Part I 11, pages 465-474. Springer, 2018.
  53. Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
  54. Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu. Combining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data. In Proceed- ings of the ACM SIGGRAPH/eurographics symposium on computer animation, pages 33-42, 2012.

Cited by

Sorted by citations

Authors
Qin, Y., Yang, W., Huang, B., Wyk, K., Su, H., Wang, X., Chao, Y.-W., & Fox, D.
Affiliations
University of California San Diego (US)
ORCID
Venue
Robotics: Science and Systems XIX
Publisher
Robotics: Science and Systems Foundation
Published
2023-07-10
DOI
10.15607/rss.2023.xix.015
OpenAlex
W4385430618
Cited by
109
References
3
Citations by year
2026: 33 · 2025: 58 · 2024: 18
FWCI
96.46
Type
conference paper
Language
en
Open access
Locations
  • Robotics: Science and Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
telerobotics, robotic hand, multi-robot teleoperation, imitation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review