CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Few-Shot In-Context Imitation Learning via Implicit Graph Alignment

Vosylius, Vitalis, Johns, Edward·2023·arXiv
DOI 10.48550/arxiv.2310.12238arXiv 2310.12238
Open this paper in CoChat — free account

Abstract

Consider the following problem: given a few demonstrations of a task across a few different objects, how can a robot learn to perform that same task on new, previously unseen objects? This is challenging because the large variety of objects within a class makes it difficult to infer the task-relevant relationship between the new objects and the objects in the demonstrations. We address this by formulating imitation learning as a conditional alignment problem between graph representations of objects. Consequently, we show that this conditioning allows for in-context learning, where a robot can perform a task on a set of new objects immediately after the demonstrations, without any prior knowledge about the object class or any further training. In our experiments, we explore and validate our design choices, and we show that our method is highly effective for few-shot learning of several real-world, everyday tasks, whilst outperforming baselines. Videos are available on our project webpage at https://www.robot-learning.uk/implicit-graph-alignment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.12238

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • [16] frames robot manipulation entirely as a series of object alignments but relies on point clouds and calibrated external cameras.

    “[16] frames robot manipulation entirely as a series of objects alignments, it however relies on point clouds and calibrated external cameras.”

    From On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation (2024) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. "Shapenet: An information-rich 3d model repository". In: arXiv preprint arXiv:1512.03012 (2015). · arXiv
  2. Y. Deng, J. Yang, and X. Tong. "Deformed implicit field: Modeling 3d shapes with learned dense correspondence". In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. 2021, pp. 10286-10296.
  3. Z. Mandi, F. Liu, K. Lee, and P. Abbeel. "Towards more generalizable one-shot visual im- itation learning". In: 2022 International Conference on Robotics and Automation (ICRA). IEEE. 2022, pp. 2434-2444.
  4. E. Johns. "Coarse-to-fine imitation learning: Robot manipulation from a single demon- stration". In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE. 2021, pp. 4613-4619.
  5. E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns. "Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning". In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE. 2022, pp. 8614-8621.
  6. P. Vitiello, K. Dreczkowski, and E. Johns. "One-Shot Imitation Learning: A Pose Esti- mation Perspective". In: Conference on Robot Learning. 2023.
  7. L. Manuelli, W. Gao, P. Florence, and R. Tedrake. "kpam: Keypoint affordances for category-level robotic manipulation". In: Robotics Research: The 19th International Symposium ISRR. Springer. 2022, pp. 132-157.
  8. W. Gao and R. Tedrake. "kpam 2.0: Feedback control for category-level robotic manip- ulation". In: IEEE Robotics and Automation Letters 6.2 (2021), pp. 2962-2969.
  9. Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. "Keto: Learning keypoint represen- tations for tool manipulation". In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE. 2020, pp. 7278-7285.
  10. P. R. Florence, L. Manuelli, and R. Tedrake. "Dense object nets: Learning dense visual object descriptors by and for robotic manipulation". In: arXiv preprint arXiv:1806.08756 (2018). · arXiv
  11. L. Manuelli, Y. Li, P. Florence, and R. Tedrake. "Keypoints into the future: Self- supervised correspondence in model-based reinforcement learning". In: arXiv preprint arXiv:2009.05085 (2020). · arXiv
  12. P. Florence, L. Manuelli, and R. Tedrake. "Self-supervised correspondence in visuomotor policy learning". In: IEEE Robotics and Automation Letters 5.2 (2019), pp. 492-499.
  13. A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. "Transporter networks: Rearranging the vi- sual world for robotic manipulation". In: Conference on Robot Learning. PMLR. 2021, pp. 726-747.
  14. V. Vosylius and E. Johns. "Where to start? Transferring simple skills to complex envi- ronments". In: Conference on Robot Learning. PMLR. 2023, pp. 471-481.
  15. M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. "Graph-structured visual imitation". In: Conference on Robot Learning. PMLR. 2020, pp. 979-989.
  16. I. Kapelyukh and E. Johns. "My house, my rules: Learning tidying preferences with graph neural networks". In: Conference on Robot Learning. PMLR. 2022, pp. 740-749.
  17. C. Wang, Y. Zhang, X. Zhang, Z. Wu, X. Zhu, S. Jin, T. Tang, and M. Tomizuka. "Offline-online learning of deformation model for cable manipulation with graph neu- ral networks". In: IEEE Robotics and Automation Letters 7.2 (2022), pp. 5544-5551.
  18. D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. "Learning models as functionals of signed-distance fields for manipulation planning". In: Conference on Robot Learning. PMLR. 2022, pp. 245-255.
  19. J.-S. Ha, D. Driess, and M. Toussaint. "Learning neural implicit functions as object rep- resentations for robotic manipulation". In: arXiv preprint arXiv:2112.04812 (2021). · arXiv
  20. J. Urain, N. Funk, G. Chalvatzaki, and J. Peters. "SE (3)-DiffusionFields: Learning cost functions for joint grasp and motion optimization through diffusion". In: arXiv preprint arXiv:2209.03855 (2022). · arXiv
  21. I. Huang, Y. Narang, R. Bajcsy, F. Ramos, T. Hermans, and D. Fox. "DefGrasp- Nets: Grasp Planning on 3D Fields with Graph Neural Nets". In: arXiv preprint arXiv:2303.16138 (2023). · arXiv
  22. A. Simeonov, Y. Du, Y.-C. Lin, A. R. Garcia, L. P. Kaelbling, T. Lozano-Pérez, and P. Agrawal. "SE (3)-Equivariant Relational Rearrangement with Neural Descriptor Fields". In: Conference on Robot Learning. PMLR. 2023, pp. 835-846.
  23. A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann. "Neural descriptor fields: Se (3)-equivariant object representations for manip- ulation". In: 2022 International Conference on Robotics and Automation (ICRA). IEEE. 2022, pp. 6394-6400.
  24. C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. "TAX-Pose: Task-Specific Cross- Pose Estimation for Robot Manipulation". In: Conference on Robot Learning. PMLR. 2023, pp. 1783-1792.
  25. C. R. Qi, L. Yi, H. Su, and L. J. Guibas. "Pointnet++: Deep hierarchical feature learning on point sets in a metric space". In: Advances in neural information processing systems 30 (2017).
  26. C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas. "Vector neurons: A general framework for so (3)-equivariant networks". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 12200-12209.
  27. L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. "Occupancy net- works: Learning 3d reconstruction in function space". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 4460-4470.
  28. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis". In: Communi- cations of the ACM 65.1 (2021), pp. 99-106.
  29. Y. Shi, Z. Huang, S. Feng, H. Zhong, W. Wang, and Y. Sun. "Masked label prediction: Unified message passing model for semi-supervised classification". In: arXiv preprint arXiv:2009.03509 (2020). · arXiv
  30. A. v. d. Oord, Y. Li, and O. Vinyals. "Representation learning with contrastive predictive coding". In: arXiv preprint arXiv:1807.03748 (2018). · arXiv
  31. Y. Du and I. Mordatch. "Implicit generation and modeling with energy based models". In: Advances in Neural Information Processing Systems 32 (2019).
  32. P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. "Implicit behavioral cloning". In: Conference on Robot Learning. PMLR. 2022, pp. 158-168.
  33. M. Welling and Y. W. Teh. "Bayesian learning via stochastic gradient Langevin dynam- ics". In: Proceedings of the 28th international conference on machine learning (ICML- 11). 2011, pp. 681-688.
  34. C. R. Qi, H. Su, K. Mo, and L. J. Guibas. "Pointnet: Deep learning on point sets for 3d classification and segmentation". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 652-660.
  35. D. Hendrycks and K. Gimpel. "Gaussian error linear units (gelus)". In: arXiv preprint arXiv:1606.08415 (2016). · arXiv
  36. I. Loshchilov and F. Hutter. "Decoupled weight decay regularization". In: arXiv preprint arXiv:1711.05101 (2017). · arXiv

Cited by

Sorted by citations

Metadata

Authors
Vosylius, V., & Johns, E.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2310.12238
arXiv
2310.12238
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review