CoChat
Sign Up
PreprintCited by 38Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

XIRL: Cross-embodiment inverse reinforcement learning

Open access
Zakka, Kevin, Zeng, Andy, Florence, Pete, Tompson, Jonathan, Bohg, Jeannette, Dwibedi, Debidatta·2021·arXiv
DOI 10.48550/arxiv.2106.03911arXiv 2106.03911
Open this paper in CoChat — free accountOpen PDF

Abstract

We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -- shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings typically required alignment with a reference trajectory, which may be difficult to acquire under stark embodiment differences. We show empirically that if the embeddings are aware of task progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. Additionally, when transferring real-world human demonstrations to a simulated robot, we find that XIRL is more sample efficient than current best methods. Qualitative results, code, and datasets are available at https://x-irl.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.03911

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning from videos of direct human demonstrations limits robot deployment because it relies on third-person visual observations and lacks real-world physical interaction data.

    “However, robot deployment is limited by its reliance on third-person visual observations and the absence of real-world physical interaction data.”

    From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [17]

    The Introduction (p. 1) describes the difficulty of learning robot policies from third-person demonstrations, including the gap between human demonstrators and robot hardware. The Conclusion (p. 8) says policy learning was demonstrated in simulation, not on a real robot. It does not identify an absence of real-world physical interaction data as the reason robot deployment is limited. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 52 · Cited by 38

Research works printed but not linked: 40

  1. [2][2] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA . IEEE, 2018. -
  2. [3][3] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE RAL , 5(2), 2019. -
  3. [4][4] P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv:1612.06699 , 2016. - · arXiv
  4. [5][5] P. Sermanet, C. Lynch, Y . Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1134-1141. IEEE, 2018. -
  5. [7][7] D. Dwibedi, Y . Aytar, J. Tompson, P . Sermanet, and A. Zisserman. Temporal cycle-consistency learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. -
  6. [8][8] C. Atkeson and S. Schaal. Robot learning from demonstration. In ICML , 1997. -
  7. [10][10] F. Torabi, G. Warnell, and P . Stone. Behavioral cloning from observation. In IJCAI , 2018. -
  8. [11][11] A. D. Edwards, H. Sahni, Y . Schroecker, and C. L. Isbell. Imitating latent policies from observation. arXiv preprint arXiv:1805.07914 , 2018. - · arXiv
  9. [12][12] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. arXiv preprint arXiv:2004.04650 , 2020. - · arXiv
  10. [13][13] D. Pathak, P . Mahmoudieh, G. Luo, P . Agrawal, D. Chen, Y . Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 2050-2053, 2018. -
  11. [14][14] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems , pages 2930-2941, 2018. -
  12. [15][15] O. Mees, M. Merklinger, G. Kalweit, and W. Burgard. Adversarial skill networks: Unsupervised robot skill learning from video. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 4188-4194. IEEE, 2020. -
  13. [17][17] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854 , 2019. - · arXiv
  14. [19][19] Y. Liu, A. Gupta, P . Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In ICRA . IEEE, 2018. -
  15. [20][20] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. In Advances in Neural Information Processing Systems , pages 2597-2607, 2019. -
  16. [21][21] D. Hejna, L. Pinto, and P . Abbeel. Hierarchically decoupled imitation for morphological transfer. In International Conference on Machine Learning , pages 4159-4171. PMLR, 2020. -
  17. [22][22] J.-Y . Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , 2017. -
  18. [23][23] P. Henderson, R. Islam, P . Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018. -
  19. [24][24] S. Toyer, R. Shah, A. Critch, and S. Russell. The magical benchmark for robust imitation. arXiv preprint arXiv:2011.00401 , 2020. - · arXiv
  20. [26][26] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. V eness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540): 529-533, 2015. -
  21. [27][27] M. Vecerik, O. Sushkov, D. Barker, T. Roth¨ orl, T. Hester, and J. Scholz. A practical approach to insertion with variable socket position using deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 754-760. IEEE, 2019. -
  22. [28][28] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 2018. - · arXiv
  23. [30][30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint:1412.6980 , 2014. - · arXiv
  24. [32][32] C. Yang, X. Ma, W. Huang, F. Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. ArXiv , abs/1910.04417, 2019. - · arXiv
  25. [34][34] G. Berseth and C. Pal. Visual imitation learning with recurrent siamese networks. ArXiv , abs/1901.07186, 2019. - · arXiv
  26. [35][35] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008. -
  27. [36][36] B. D. Ziebart, J. A. Bagnell, and A. K. Dey. Modeling interaction via the principle of maximum causal entropy. In ICML , 2010. -
  28. [37][37] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 , 2014. - · arXiv
  29. [39][39] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158 , 2018. - · arXiv
  30. [40][40] B. C. Stadie, P . Abbeel, and I. Sutskever. Third-person imitation learning. arXiv:1703.01703 , 2017. - · arXiv
  31. [41][41] Y. Lu and J. Tompson. Adail: Adaptive adversarial imitation learning. NeurIPS Workshop on Learning Transferable Skills, , 2019. -
  32. [42][42] F. Liu, Z. Ling, T. Mu, and H. Su. State alignment-based imitation learning. arXiv preprint arXiv:1911.10947 , 2019. - · arXiv
  33. [43][43] W. Sun, A. Vemula, B. Boots, and J. Bagnell. Provably efficient imitation learning from observation alone. ArXiv , abs/1905.10948, 2019. - · arXiv
  34. [44][44] K. Kim, Y . Gu, J. Song, S. Zhao, and S. Ermon. Domain adaptive imitation learning. arXiv , 2020. -
  35. [45][45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 . Curran Associates, Inc., 2019. -
  36. [46][46] A. Buslaev, A. Parinov, E. Khvedchenya, V . Iglovikov, and A. A. Kalinin. Albumentations: fast and flexible image augmentations. arXiv e-prints , 2018. -
  37. [48][48] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. arXiv e-prints , 2015. -
  38. [49][49] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018 , 2018. -
  39. [50][50] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv e-prints , 2013. -
  40. [51][51] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 , 2020. · arXiv

Authors
Zakka, K., Zeng, A., Florence, P., Tompson, J., Bohg, J., & Dwibedi, D.
Affiliations
  • University of California, Berkeley (US)
  • Google (United States) (US)
  • Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.03911
arXiv
2106.03911
OpenAlex
W3167325792
Cited by
38
References
52
Citations by year
2025: 5 · 2024: 14 · 2023: 14 · 2022: 3 · 2021: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
inverse reinforcement learning, self-supervised learning, image embeddings, human demonstrations, sample efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review