CoChat
Sign Up
PreprintCited by 3Robot Manipulation and LearningReinforcement Learning in RoboticsSocial Robot Interaction and HRI

Okami: Teaching humanoid robots manipulation skills through single video imitation

Open access
Li, Jinhan, Zhu, Yifeng, Xie, Yuqi, Jiang, Zhenyu, Seo, Mingyo, Pavlakos, Georgios, Zhu, Yuke·2024·arXiv
DOI 10.48550/arxiv.2410.11792arXiv 2410.11792
Open this paper in CoChat — free accountOpen PDF

Abstract

We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of 79.2% without the need for labor-intensive teleoperation. More videos can be found on our website https://ut-austin-rpl.github.io/OKAMI/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.11792

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning from videos of direct human demonstrations limits robot deployment because it relies on third-person visual observations and lacks real-world physical interaction data.

    “However, robot deployment is limited by its reliance on third-person visual observations and the absence of real-world physical interaction data.”

    From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [21]

    The Introduction (p. 2) says learning from a human video is challenging because the video lacks action labels, supporting the claim that visual observation has limitations. It does not identify a lack of real-world physical interaction data as a limit on deployment; Section 4.3 (p. 8) describes collecting robot rollout trajectories to train visuomotor policies. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 62

  1. [1][1] M. Seo, S. Han, K. Sim, S. H. Bang, C. Gonzalez, L. Sentis, and Y. Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation. In IEEE-RAS International Conference on Humanoid Robots (Humanoids) , 2023. -
  2. [2][2] Y. Matsuura, K. Kawaharazuka, N. Hiraoka, K. Kojima, K. Okada, and M. Inaba. Development of a whole-body work imitation learning system by a biped and bi-armed humanoid. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1037410381. IEEE, 2023. -
  3. [3][3] T. Asfour, P. Azad, F. Gyarfas, and R. Dillmann. Imitation learning of dual-arm manipulation tasks in humanoid robots. International journal of humanoid robotics , 5(02):183-202, 2008. -
  4. [4][4] Y. Zhu, A. Lim, P. Stone, and Y. Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint arXiv:2405.20321 , 2024. - · arXiv
  5. [5][5] N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada. Ditto: Demonstration imitation by trajectory transformation. arXiv preprint arXiv:2403.15203 , 2024. - · arXiv
  6. [6][6] D. Guo. Learning multi-step manipulation tasks from a single human demonstration. arXiv preprint arXiv:2312.15346 , 2023. - · arXiv
  7. [7][7] T. Asfour and R. Dillmann. Human-like motion of a humanoid robot arm based on a closedform solution of the inverse kinematics problem. In Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453) , volume 2, pages 1407-1412. IEEE, 2003. -
  8. [8][8] M. Gleicher. Retargetting motion to new characters. Proceedings of the 25th annual conference on Computer graphics and interactive techniques , 1998. -
  9. [9][9] K. Darvish, Y. Tirupachuri, G. Romualdi, L. Rapetti, D. Ferigo, F. J. A. Chavez, and D. Pucci. Whole-body geometric retargeting for humanoid robots. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids) , pages 679-686. IEEE, 2019. -
  10. [10][10] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796 , 2024. - · arXiv
  11. [11][11] S. Nakaoka, A. Nakazawa, F. Kanehiro, K. Kaneko, M. Morisawa, and K. Ikeuchi. Task model of lower body motion for a biped humanoid robot to imitate human dances. In 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 3157-3162. IEEE, 2005. -
  12. [12][12] K. Hu, C. Ott, and D. Lee. Online human walking imitation in task and joint space based on quadratic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 3458-3464. IEEE, 2014. -
  13. [14][14] E. Demircan, T. Besier, S. Menon, and O. Khatib. Human motion reconstruction and synthesis of human skills. In Advances in Robot Kinematics: Motion in Man and Machine: Motion in Man and Machine , pages 283-292. Springer, 2010. -
  14. [16][16] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023. - · arXiv
  15. [17][17] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-humanoid real-time whole-body teleoperation. In arXiv , 2024. -
  16. [18][18] A. Escande, N. Mansard, and P.-B. Wieber. Hierarchical quadratic programming: Fast online humanoid-robot motion generation. The International Journal of Robotics Research , 33(7): 1006-1028, 2014. -
  17. [19][19] Q. Liao, B. Zhang, X. Huang, X. Huang, Z. Li, and K. Sreenath. Berkeley humanoid: A research platform for learning-based control. arXiv preprint arXiv:2407.21781 , 2024. - · arXiv
  18. [20][20] L. Penco, N. Scianca, V. Modugno, L. Lanari, G. Oriolo, and S. Ivaldi. A multimode teleoperation framework for humanoid loco-manipulation: An application for the icub robot. IEEE Robotics & Automation Magazine , 26(4):73-82, 2019. -
  19. [21][21] D. Kim, B.-J. You, and S.-R. Oh. Whole body motion control framework for arbitrarily and simultaneously assigned upper-body tasks and walking motion. Modeling, Simulation and Optimization of Bipedal Walking , pages 87-98, 2013. -
  20. [22][22] A. Di Fava, K. Bouyarmane, K. Chappellet, E. Ruffaldi, and A. Kheddar. Multi-contact motion retargeting from human to humanoid robot. In 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids) , pages 1081-1086. IEEE, 2016. -
  21. [23][23] M. Arduengo, A. Arduengo, A. Colom´ e, J. Lobo-Prat, and C. Torras. Human to robot wholebody motion transfer. In 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids) , pages 299-305. IEEE, 2021. -
  22. [24][24] R. Cisneros, M. Benallegue, K. Kaneko, H. Kaminaga, G. Caron, A. Tanguy, R. Singh, L. Sun, A. Dallard, C. Fournier, et al. Team janus humanoid avatar: A cybernetic avatar to embody human telepresence. In Toward Robot Avatars: Perspectives on the ANA Avatar XPRIZE Competition, RSS Workshop , volume 3, 2022. -
  23. [25][25] S. Tachi, K. Komoriya, K. Sawada, T. Nishiyama, T. Itoko, M. Kobayashi, and K. Inoue. Telexistence cockpit for humanoid robot control. Advanced Robotics , 17(3):199-217, 2003. -
  24. [26][26] J. Ramos and S. Kim. Humanoid dynamic synchronization through whole-body bilateral feedback teleoperation. IEEE Transactions on Robotics , 34(4):953-965, 2018. -
  25. [27][27] Y. Ishiguro, T. Makabe, Y. Nagamatsu, Y. Kojio, K. Kojima, F. Sugai, Y. Kakiuchi, K. Okada, and M. Inaba. Bilateral humanoid teleoperation system using whole-body exoskeleton cockpit tablis. IEEE Robotics and Automation Letters , 5(4):6419-6426, 2020. -
  26. [29][29] M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke. Nimbro avatar: Interactive immersive telepresence with force-feedback telemanipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5312-5319. IEEE, 2021. -
  27. [30][30] M. Hirschmanner, C. Tsiourti, T. Patten, and M. Vincze. Virtual reality teleoperation of a humanoid robot using markerless human upper body pose imitation. in 2019 ieee-ras 19th international conference on humanoid robots (humanoids), 2019. -
  28. [31][31] D. Lim, D. Kim, and J. Park. Online telemanipulation framework on humanoid for both manipulation and imitation. 2022 19th International Conference on Ubiquitous Robots (UR) , pages 8-15, 2022. URL https://api.semanticscholar.org/CorpusID:250577582 . -
  29. [32][32] Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454 , 2024. - · arXiv
  30. [33][33] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart´ ın-Mart´ ın. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 , 2021. - · arXiv
  31. [34][34] A. Mandlekar, D. Xu, R. Mart´ ın-Mart´ ın, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085 , 2020. - · arXiv
  32. [35][35] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422 , 2023. - · arXiv
  33. [36][36] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11339 , 2022. - · arXiv
  34. [37][37] C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788 , 2024. - · arXiv
  35. [38][38] T. Lin, Y. Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and J. Malik. Learning visuotactile skills with two multifingered hands. arXiv preprint arXiv:2404.16823 , 2024. - · arXiv
  36. [39][39] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy. arXiv preprint arXiv:2403.03954 , 2024. - · arXiv
  37. [40][40] M. Chang and S. Gupta. One-shot visual imitation via attributed waypoints and demonstration augmentation. arXiv preprint arXiv:2302.04856 , 2023. - · arXiv
  38. [41][41] T. Yu, P. Abbeel, S. Levine, and C. Finn. One-shot composition of vision-based skills from demonstration. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2643-2650. IEEE, 2019. -
  39. [42][42] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns. Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8614-8621. IEEE, 2022. -
  40. [43][43] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA) , pages 4613-4619. IEEE, 2021. -
  41. [44][44] N. Di Palo and E. Johns. Learning multi-stage tasks with one demonstration via self-replay. In Conference on Robot Learning , pages 1180-1189. PMLR, 2022. -
  42. [45][45] Z. Luo, J. Cao, K. Kitani, W. Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10895-10904, 2023. -
  43. [46][46] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG) , 40(4): 1-20, 2021. -
  44. [47][47] B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems , 36, 2024. -
  45. [49][49] Y. Liang, W. Li, Y. Wang, R. Xiong, Y. Mao, and J. Zhang. Dynamic movement primitive based motion retargeting for dual-arm sign language motions. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 8195-8201. IEEE, 2021. -
  46. [50][50] S. Caelles, J. Pont-Tuset, F. Perazzi, A. Montes, K.-K. Maninis, and L. Van Gool. The 2019 davis challenge on vos: Unsupervised multi-object segmentation. arXiv preprint arXiv:1905.00737 , 2019. - · arXiv
  47. [51][51] Y. Huang, J. Yuan, C. Kim, P. Pradhan, B. Chen, L. Fuxin, and T. Hermans. Out of sight, still in mind: Reasoning and planning about unobserved objects with video tracking enabled memory models. arXiv preprint arXiv:2309.15278 , 2023. - · arXiv
  48. [52][52] Y. Zhu, Z. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. In 7th Annual Conference on Robot Learning , 2023. -
  49. [53][53] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, S. Kirmani, B. Zitkovich, F. Xia, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905 , 2023. - · arXiv
  50. [54][54] H. K. Cheng, S. W. Oh, B. Price, J.-Y. Lee, and A. Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3151-3161, 2024. -
  51. [55][55] V. Ye, G. Pavlakos, J. Malik, and A. Kanazawa. Decoupling human and camera motion from videos in the wild. In CVPR , 2023. -
  52. [56][56] J. Romero, D. Tzionas, and M. J. Black. Embodied hands. ACM Transactions on Graphics , 36 (6):1-17, 2017. -
  53. [57][57] G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3D with transformers. In CVPR , 2024. -
  54. [58][58] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635 , 2023. - · arXiv
  55. [59][59] Q.-Y. Zhou, J. Park, and V. Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv:1801.09847 , 2018. - · arXiv
  56. [60][60] Y. Zhu, J. Wong, A. Mandlekar, R. Mart´ ın-Mart´ ın, A. Joshi, S. Nasiriany, and Y. Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 , 2020. - · arXiv
  57. [61][61] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023. - · arXiv
  58. [62][62] S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa, and J. Malik. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV , 2023. -
  59. [63][63] Y. Xu, J. Zhang, Q. Zhang, and D. Tao. ViTPose++: Vision transformer for generic body pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. -
  60. [66][66] R. Killick, P. Fearnhead, and I. A. Eckley. Optimal detection of changepoints with a linear computational cost. Journal of the American Statistical Association , 107(500):1590-1598, 2012. -
  61. [67][67] X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immersive active visual feedback. arXiv preprint arXiv:2407.01512 , 2024. - · arXiv
  62. [68][68] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588 , 2023. - · arXiv

Cited by

Sorted by citations

Related

Authors
Li, J., Zhu, Y., Xie, Y., Jiang, Z., Seo, M., Pavlakos, G., & Zhu, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.11792
arXiv
2410.11792
OpenAlex
W4403575288
Cited by
3
References
0
Citations by year
2025: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
humanoid robots, manipulative skills, RGB-D video, visuomotor policies, human motion imitation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review