CoChat
Sign Up
Cited by 186Robot Manipulation and LearningSocial Robot Interaction and HRITeleoperation and Haptic Systems

Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Open access
Chi, Cheng, Xu, Zhenjia, Pan, Chuer, Cousineau, Eric, Burchfiel, Benjamin, Feng, Siyuan, Tedrake, Russ, Song, Shuran·2024·Robotics: Science and Systems XX
DOI 10.15607/rss.2024.xx.045
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.15607/rss.2024.xx.045

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Handheld tools for demonstration still need a system-specific tool or modifications to the robot's original hardware to align the end-effector's physical interactions with the recorded visual observations.

    “While these approaches simplify tool design, they still require a system-specific tool or modifications to a robot's original hardware to align the end-effector's physical interactions and the recorded visual observations.”

    From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [15]

    The Demonstration Interface Design section (page unavailable) says UMI places a GoPro at the same position relative to the same gripper fingers during demonstration and robot deployment to minimize the visual observation gap. The paper also says robot arms can be equipped with a compatible gripper and camera setup. It does not establish that handheld approaches generally require a system-specific tool or modifications to a robot’s original hardware. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 42

  1. [1]1. Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  2. [2]2. Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778-13790, 2023.
  3. [3]3. Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  4. [4]4. Anais Brygo, Ioannis Sarakoglou, Nadia Garcia- Hernandez, and Nikolaos Tsagarakis. Humanoid robot teleoperation with vibrotactile based balancing feedback. In Haptics: Neuroscience, Devices, Modeling, and Ap- plications: 9th International Conference, EuroHaptics 2014, Versailles, France, June 24-26, 2014, Proceedings, Part II 9, pages 266-275. Springer, 2014.
  5. [6]6. Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874-1890, 2021.
  6. [7]7. Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, and Juan D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874-1890, 2021. doi: 10.1109/TRO. 2021.3075644. · DOI
  7. [8]8. Annie S Chen, Suraj Nair, and Chelsea Finn. Learn- ing generalizable robotic reward functions from "in-the- wild" human videos. In Proceedings of Robotics: Science and Systems (RSS), 2021.
  8. [9]9. Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- fusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  9. [11]11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  10. [13]13. Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Dani- ilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross- domain datasets. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  11. [16]16. S. Garrido-Jurado, R. Muñoz-Salinas, F.J. Madrid- Cuevas, and M.J. Marín-Jiménez. Automatic genera- tion and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6):2280-2292, 2014. ISSN 0031-3203. doi: https://doi.org/10.1016/ j.patcog.2014.01.005. URL https://www.sciencedirect. com/science/article/pii/S0031320314000235.
  12. [19]19. Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), volume 164, pages 991-1002. PMLR, 2022.
  13. [20]20. Moo Jin Kim, Jiajun Wu, and Chelsea Finn. Giv- ing robots a hand: Broadening generalization via hand- centric human video demonstrations. In Deep Reinforce- ment Learning Workshop NeurIPS, 2022.
  14. [21]21. Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, 2023.
  15. [22]22. Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning (CoRL), volume 87, pages 879-893. PMLR, 2018.
  16. [23]23. Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In Proceedings of The 6th Conference on Robot Learning (CoRL), volume 205, pages 892-909. PMLR, 2022.
  17. [24]24. Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In Proceedings of The 6th Conference on Robot Learning (CoRL), volume 205, pages 1783-1792. PMLR, 2023.
  18. [25]25. Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pan- dian Arunachalam, and Lerrel Pinto. The surprising ef- fectiveness of representation learning for visual imitation. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  19. [26]26. Luka Peternel and Jan Babič. Learning of compliant human-robot interaction using full-body haptic interface. Advanced Robotics, 27(13):1003-1012, 2013.
  20. [27]27. Pragathi Praveena, Guru Subramani, Bilge Mutlu, and Michael Gleicher. Characterizing input methods for human-to-robot demonstrations. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 344-353. IEEE, 2019.
  21. [28]28. Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Im- itation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570-587. Springer, 2022.
  22. [29]29. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learning, pages 8748-8763. PMLR, 2021.
  23. [30]30. Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3:297-330, 2020.
  24. [31]31. Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task- agnostic offline reinforcement learning. In Proceedings of The 6th Conference on Robot Learning (CoRL), vol- ume 205, pages 1838-1849. PMLR, 2023.
  25. [32]32. Felipe Sanches, Geng Gao, Nathan Elangovan, Ricardo V Godoy, Jayden Chapman, Ke Wang, Patrick Jarvis, and Minas Liarokapis. Scalable. intuitive human to robot skill transfer with wearable human machine interfaces: On complex, dexterous tasks. In 2023 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 6318-6325. IEEE, 2023.
  26. [33]33. Karl Schmeckpeper, Annie Xie, Oleh Rybkin, Stephen Tian, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pages 708-725. Springer, 2020.
  27. [34]34. Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Reinforcement learn- ing with videos: Combining offline observations with interaction. In Proceedings of the 2020 Conference on Robot Learning (CoRL), volume 155, pages 339-354. PMLR, 2021.
  28. [35]35. Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Hu- manoids), pages 1-8. IEEE, 2023.
  29. [37]37. Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning manipu- lation concepts from instructions and human demonstra- tions. The International Journal of Robotics Research, 40(12-14):1419-1434, 2021.
  30. [38]38. Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Proceedings of The 6th Conference on Robot Learning (CoRL), volume 205, pages 654-665. PMLR, 2023.
  31. [39]39. William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation. In Proceedings of The 7th Conference on Robot Learning (CoRL), volume 229, pages 405-424. PMLR, 2023.
  32. [40]40. Anthony Simeonov, Yilun Du, Andrea Tagliasac- chi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipula- tion. In 2022 International Conference on Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
  33. [41]41. Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. Robotics and Automation Letters, 2020.
  34. [43]43. Alexander Toedtheide, Xiao Chen, Hamid Sadeghian, Abdeldjallil Naceri, and Sami Haddadin. A force- sensitive exoskeleton for teleoperation: An application in elderly care robotics. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12624-12630. IEEE, 2023.
  35. [44]44. Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anand- kumar. Mimicplay: Long-horizon imitation learning by watching human play. In Proceedings of The 7th Conference on Robot Learning (CoRL), volume 229, pages 201-221. PMLR, 2023.
  36. [46]46. Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Man- dlekar, Li Fei-Fei, Silvio Savarese, and Roberto Martín- Martín. Error-aware imitation learning from teleoperation data for mobile manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), volume 164, pages 1367-1378. PMLR, 2022.
  37. [48]48. Keenan A Wyrobek, Eric H Berger, HF Machiel Van der Loos, and J Kenneth Salisbury. Towards a personal robotics development platform: Rationale and design of an intrinsically safe personal robot. In 2008 IEEE International Conference on Robotics and Automation, pages 2165-2170. IEEE, 2008.
  38. [50]50. Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha, and Animesh Garg. Learn- ing by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
  39. [51]51. Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy. In Conference on Robot Learning (CoRL), volume 155, pages 1992-2005. PMLR, 2021.
  40. [52]52. Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018.
  41. [54]54. Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  42. [55]55. Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. In Proceedings of The 6th Conference on Robot Learning (CoRL), volume 205, pages 1199-1210. PMLR, 2023.

Cited by

Sorted by citations

Related

Authors
Chi, C., Xu, Z., Pan, C., Cousineau, E., Burchfiel, B., Feng, S., Tedrake, R., & Song, S.
Affiliations
  • Columbia University (US)
  • Stanford University (US)
  • Toyota Research Institute (US)
ORCID
Venue
Robotics: Science and Systems XX
Publisher
Robotics: Science and Systems Foundation
Published
2024-07-15
DOI
10.15607/rss.2024.xx.045
OpenAlex
W4402354127
Cited by
183
References
0
Citations by year
2026: 62 · 2025: 107 · 2024: 14
FWCI
123.67
Type
conference paper
Language
en
Open access
Locations
Robotics: Science and Systems (publishedVersion)
Funders
  • National Science Foundation (2037101, 2132519)
  • Toyota Research Institute
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
Keywords
robot teaching, human-robot interaction, robot learning from demonstration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review