CoChat
Sign Up
PreprintCited by 11Robot Manipulation and LearningReinforcement Learning in RoboticsDomain Adaptation and Few-Shot Learning

XSkill: Cross embodiment skill discovery

Open access
Xu, Mengda, Xu, Zhenjia, Chi, Cheng, Veloso, Manuela, Song, Shuran·2023·arXiv
DOI 10.48550/arxiv.2307.09955arXiv 2307.09955
Open this paper in CoChat — free accountOpen PDF

Abstract

Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework. The benchmark, code, and qualitative results are on https://xskill.cs.columbia.edu/

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.09955

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning from videos of direct human demonstrations limits robot deployment because it relies on third-person visual observations and lacks real-world physical interaction data.

    “However, robot deployment is limited by its reliance on third-person visual observations and the absence of real-world physical interaction data.”

    From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [20]

    The Abstract says extracting robot skills directly from human videos is challenging because of embodiment differences and unobserved action parameters. The Additional information section describes human videos as RGB observations only, but also describes real-world robot teleoperation data with recorded actions. The paper does not attribute limits on robot deployment to an absence of real-world physical interaction data or specifically to third-person views. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 11

Research works printed but not linked: 57

  1. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
  2. Y. M. Asano, C. Rupprecht, and A. Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.05371, 2019. · arXiv
  3. M. A. Bautista, A. Sanakoyeu, E. Tikhoncheva, and B. Ommer. Cliquecnn: Deep unsupervised exemplar learning. Advances in Neural Information Processing Systems, 29, 2016.
  4. M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
  5. A. Bagaria and G. Konidaris. Option discovery using deep skill chaining. In International Con- ference on Learning Representations, 2020. URL https://openreview.net/forum? id=B1gqipNYwH.
  6. B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. ArXiv, abs/1802.06070, 2018. · arXiv
  7. A. Sharma, S. S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. ArXiv, abs/1907.01657, 2019. · arXiv
  8. R. Fox, S. Krishnan, I. Stoica, and K. Goldberg. Multi-level discovery of deep options. ArXiv, abs/1703.08294, 2017. · arXiv
  9. G. D. Konidaris and A. G. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In NIPS, 2009.
  10. V. C. V. Kumar, S. Ha, and C. K. Liu. Expanding motor skills using relay networks. In Conference on Robot Learning, 2018.
  11. K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. ArXiv, abs/1611.07507, 2016. · arXiv
  12. K. Hausman, J. T. Springenberg, Z. Wang, N. M. O. Heess, and M. A. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
  13. T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In Pro- ceedings of (ICML) International Conference on Machine Learning, pages 8624 -8633, July 2020.
  14. L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient ex- ploration via state marginal matching, 2020. URL https://openreview.net/forum? id=Hkla1eHFvS.
  15. M. Laskin, H. Liu, X. B. Peng, D. Yarats, A. Rajeswaran, and P. Abbeel. Un- supervised reinforcement learning with contrastive intrinsic control. 35:34478-34491, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/debf482a7dbdc401f9052dbe15702837-Paper-Conference.pdf.
  16. D. Tanneberg, K. Ploeger, E. Rueckert, and J. Peters. SKID RAW: skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 6(3):4696-4703, 2021. doi:10.1109/LRA. 2021.3068891. · DOI
  17. A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=Ysuv-WOFeKR.
  18. K. Pertsch, Y. Lee, and J. J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
  19. M. Xu, M. Veloso, and S. Song. ASPire: Adaptive skill priors for reinforcement learning. In Advances in Neural Information Processing Systems, 2022. URL https://openreview. net/forum?id=sr0289wAUa.
  20. Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 7:4126-4133, 2021.
  21. A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart'in-Mart'in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, 2021.
  22. P. R. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5:492-499, 2019.
  23. R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task ma- nipulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758-3765, 2017.
  24. A. Zeng, P. R. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, 2020.
  25. T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, 2017.
  26. N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=agTr-vRQsa.
  27. Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. VIOLA: Object-centric imitation learning for vision- based robot manipulation. In 6th Annual Conference on Robot Learning, 2022. URL https: //openreview.net/forum?id=L8hCfhPbFho.
  28. A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt- 1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2204.01691, 2022. · arXiv
  29. P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning (CoRL), 2021.
  30. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023. · arXiv
  31. T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Pv1GPQzRrC8.
  32. M. Reuss, M. X. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score- based diffusion policies. ArXiv, abs/2304.02532, 2023. · arXiv
  33. C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2019.
  34. A. Mandlekar, F. Ramos, B. Boots, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforce- ment without interaction at scale for learning control from offline robot manipulation data. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420, 2019.
  35. A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. ArXiv, abs/2003.06085, 2020. · arXiv
  36. K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner. Taco: Learning task decompo- sition via temporal alignment for control. In International Conference on Machine Learning, 2018.
  37. Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. In NIPS, 2017.
  38. Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. 2022 International Conference on Robotics and Automation (ICRA), pages 2434- 2444, 2021.
  39. C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. ArXiv, abs/1709.04905, 2017. · arXiv
  40. S. Dasari and A. K. Gupta. Transformers for one-shot visual imitation. ArXiv, abs/2011.05970, 2020. · arXiv
  41. T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. ArXiv, abs/2203.06173, 2022. · arXiv
  42. K. Zakka, A. Zeng, P. R. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross- embodiment inverse reinforcement learning. In Conference on Robot Learning, 2021.
  43. L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):1419-1434, 2021.
  44. H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834, 2021.
  45. K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. CoRR, abs/2011.06507, 2020. URL https://arxiv.org/abs/2011.06507. · arXiv
  46. P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. CoRR, abs/1612.06699, 2016. URL http://arxiv.org/abs/1612.06699. · arXiv
  47. A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos. ArXiv, abs/2103.16817, 2021. · arXiv
  48. M. Sieb, X. Zhou, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. In Conference on Robot Learning, 2019.
  49. S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang. Graph inverse reinforcement learning from diverse videos. In Conference on Robot Learning, 2022.
  50. S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. ArXiv, abs/2207.09450, 2022. · arXiv
  51. C. Wang, L. J. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mim- icplay: Long-horizon imitation learning by watching human play. ArXiv, abs/2302.12422, 2023. · arXiv
  52. M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
  53. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  54. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
  55. I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020. · arXiv
  56. A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. · arXiv
  57. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006.11239, 2020. · arXiv

Authors
Xu, M., Xu, Z., Chi, C., Veloso, M., & Song, S.
Affiliations
  • JPMorgan Chase & Co (United States) (US)
  • Columbia University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.09955
arXiv
2307.09955
OpenAlex
W4384919635
Cited by
11
References
0
Citations by year
2025: 5 · 2024: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2143601, 2037101, 2132519)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
imitation learning, skill transfer, skill composition, robotic manipulation, embodied AI, unseen tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review