CoChat
Sign Up
PreprintCited by 94Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

The Surprising Effectiveness of Representation Learning for Visual Imitation

Open access
Pari, Jyothish, Shafiullah, Nur Muhammad, Arunachalam, Sridhar Pandian, Pinto, Lerrel·2021·arXiv
DOI 10.48550/arxiv.2112.01511arXiv 2112.01511
Open this paper in CoChat — free account

Abstract

While visual imitation learning offers one of the most effective ways of learning from visual demonstrations, generalizing from them requires either hundreds of diverse demonstrations, task specific priors, or large, hard-to-train parametric models. One reason such complexities arise is because standard visual imitation frameworks try to solve two coupled problems at once: learning a succinct but good representation from the diverse visual data, while simultaneously learning to associate the demonstrated actions with such representations. Such joint learning causes an interdependence between these two problems, which often results in needing large amounts of demonstrations for learning. To address this challenge, we instead propose to decouple representation learning from behavior learning for visual imitation. First, we learn a visual representation encoder from offline data using standard supervised and self-supervised learning methods. Once the representations are trained, we use non-parametric Locally Weighted Regression to predict the actions. We experimentally show that this simple decoupling improves the performance of visual imitation models on both offline demonstration datasets and real-robot door opening compared to prior work in visual imitation. All of our generated data, code, and robot videos are publicly available at https://jyopari.github.io/VINN/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2112.01511

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • k-NN finds trajectories in the dataset that are close, via distance in feature space, to the goal image.

    “k -NN [87] finds trajectories in the dataset that are close (via distance in feature space [84]) to the goal image.”

    From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [87]

    In “k-Nearest Neighbors Based Locally Weighted Regression,” the paper describes using feature-space distance to find demonstration frames similar to an input image. It does not describe finding whole trajectories close to a goal image; the query is the current observation, and the retrieved frames’ actions are averaged. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. David W. Aha and Steven L. Salzberg. Learning to catch: Applying nearest neighbor algorithms to dynamic control tasks. In P. Cheeseman and R. W. Oldford, editors, Selecting Models from Data, pages 321-328, New York, NY, 1994. Springer New York. ISBN 978-1-4612-2660- 4.
  2. Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57 (5):469-483, 2009.
  3. Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, pages 11-73, 1997.
  4. Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self- supervised learning. arXiv preprint arXiv:2105.04906, 2021. · arXiv
  5. Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assign- ments. arXiv preprint arXiv:2006.09882, 2020. · arXiv
  6. Bryan Chen, Alexander Sax, Gene Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representa- tions: An experimental study in manipulation and navi- gation. arXiv preprint arXiv:2011.06698, 2020. · arXiv
  7. Ting Chen, Simon Kornblith, Kevin Swersky, Moham- mad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020. · arXiv
  8. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. · arXiv
  9. Yan Duan, Marcin Andrychowicz, Bradly Stadie, Ope- nAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pages 1087-1098, 2017.
  10. Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a lit- tle help from my friends: Nearest-neighbor contrastive learning of visual representations. arXiv preprint arXiv:2104.14548, 2021. · arXiv
  11. Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. arXiv preprint arXiv:2109.00137, 2021. · arXiv
  12. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. · arXiv
  13. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self- supervised learning. arXiv preprint arXiv:2006.07733, 2020. · arXiv
  14. Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cos- min Paduraru, et al. Rl unplugged: Benchmarks for offline reinforcement learning. arXiv e-prints, pages arXiv-2006, 2020.
  15. Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50 (2):1-35, 2017.
  16. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion- scale similarity search with gpus. arXiv preprint arXiv:1702.08734, 2017. · arXiv
  17. Charles C Kemp, Aaron Edsinger, Henry M Clever, and Blaine Matulevich. The design of stretch: A compact, lightweight mobile manipulator for indoor human envi- ronments. arXiv preprint arXiv:2109.10892, 2021. · arXiv
  18. Elman Mansimov and Kyunghyun Cho. Simple near- est neighbor policy method for continuous control tasks, 2018. URL https://openreview.net/forum?id= ByL48G-AW.
  19. Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.05085, 2020. · arXiv
  20. Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin'ichi Satoh. A survey of product quantization. ITE Transactions on Media Technology and Applications, 6 (1):2-10, 2018.
  21. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zem- ing Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019.
  22. Jean Piaget. Play, dreams and imitation in childhood, volume 25. Routledge, 2013.
  23. Stephane Ross, Geoffrey Gordan, and Andrew Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In arXiv preprint arXiv:1011.0686, 2010. · arXiv
  24. Alexander Sax, Jeffrey O Zhang, Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, and Jitendra Malik. Learning to navigate using mid-level visual priors. arXiv preprint arXiv:1912.11121, 2019. · arXiv
  25. Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsu- pervised perceptual rewards for imitation learning. arXiv preprint arXiv:1612.06699, 2016. · arXiv
  26. Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3):4978-4985, 2020.
  27. Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. ICLR, 2017.
  28. Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from re- inforcement learning. In International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
  29. Michael Tomasello, Sue Savage-Rumbaugh, and Ann Cale Kruger. Imitative learning of actions on objects by children, chimpanzees, and enculturated chimpanzees. Child development, 64(6):1688-1705, 1993.
  30. Faraz Torabi, Garrett Warnell, and Peter Stone. Be- havioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018. · arXiv
  31. Yusuke Urakami, Alec Hodgkinson, Casey Carlin, Ran- dall Leu, Luca Rigazio, and Pieter Abbeel. Doorgym: A scalable door opening environment and baseline agent. CoRR, abs/1908.01887, 2019. URL http://arxiv.org/abs/ 1908.01887. · arXiv
  32. Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021. · arXiv
  33. Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical repre- sentations. arXiv preprint arXiv:2102.11271, 2021. · arXiv
  34. Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy. arXiv e-prints, pages arXiv-2008, 2020.
  35. Sarah Young, Jyothish Pari, Pieter Abbeel, and Lerrel Pinto. Playful interactions for representation learning. arXiv preprint arXiv:2107.09046, 2021. · arXiv
  36. Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. arXiv preprint arXiv:2012.07975, 2020. · arXiv
  37. Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
  38. Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Rein- forcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564, 2018. · arXiv
  39. Yuke Zhu, Ziyu Wang, Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, abs/1802.09564, 2018. URL http://arxiv.org/abs/1802.09564. · arXiv

Cited by

Sorted by citations

Authors
Pari, J., Shafiullah, N. M., Arunachalam, S. P., & Pinto, L.
Affiliations
New York University (US)
Corresponding author
Pinto, Lerrel
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2112.01511
arXiv
2112.01511
OpenAlex
W3214952050
Cited by
94
References
1
Citations by year
2026: 13 · 2025: 35 · 2024: 27 · 2023: 15 · 2022: 4
FWCI
11.20
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • Robotics: Science and Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Office of Naval Research (N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
visual imitation learning, representation learning, self-supervised learning, locally weighted regression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review