The Surprising Effectiveness of Representation Learning for Visual Imitation
Open accessAbstract
While visual imitation learning offers one of the most effective ways of learning from visual demonstrations, generalizing from them requires either hundreds of diverse demonstrations, task specific priors, or large, hard-to-train parametric models.One reason such complexities arise is because standard visual imitation frameworks try to solve two coupled problems at once: learning a succinct but good representation from the diverse visual data, while simultaneously learning to associate the demonstrated actions with such representations.Such joint learning causes an interdependence between these two problems, which often results in needing large amounts of demonstrations for learning.To address this challenge, we instead propose to decouple representation learning from behavior learning for visual imitation.First, we learn a visual representation encoder from offline data using standard supervised and self-supervised learning methods.Once the representations are trained, we use non-parametric Locally Weighted Regression to predict the actions.We experimentally show that this simple decoupling improves the performance of visual imitation models on both offline demonstration datasets and real-robot door opening compared to prior work in visual imitation.All of our generated data, code, and robot videos are publicly available at https://jyopari.github.io/VINN/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Handheld tools for demonstration still need a system-specific tool or modifications to the robot's original hardware to align the end-effector's physical interactions with the recorded visual observations.
“While these approaches simplify tool design, they still require a system-specific tool or modifications to a robot's original hardware to align the end-effector's physical interactions and the recorded visual observations.”
From LEGATO: Cross-Embodiment Imitation Using a Grasping Tool (2025) · cites this paper as [13]
The Robot details section (page unavailable) describes a handheld reacher-grabber fitted with a camera and modifications to both its grip and the robot’s gripper. It says the gripper changes were made for easier resets, not to align physical interactions with visual observations. The paper documents those changes in its own setup but does not establish that handheld demonstration approaches generally require them. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Prototypical Networks for Few-shot LearningPreprint
Snell, J., Swersky, K., & Zemel, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05175 · Cited by 5,863 · see citing works
Play, Dreams And Imitation In Childhood
Piaget, J. (2013) · Routledge eBooks · doi:10.4324/9781315009698 · Cited by 5,416 · see citing works
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · see citing works
Imitation of Facial and Manual Gestures by Human Neonates
Meltzoff, A. N., & Moore, M. (1977) · Science · doi:10.1126/science.198.4312.75 · Cited by 3,277 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · see citing works
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
Atkeson, C. G., Moore, A., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006559212014 · Cited by 1,684 · see citing works
Big Self-Supervised Models are Strong Semi-Supervised LearnersPreprint
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 1,480 · see citing works
Discriminative Unsupervised Feature Learning with Convolutional Neural NetworksPreprint
Dosovitskiy, A., Springenberg, J., Riedmiller, M. A., & Brox, T. (2014) · Neural Information Processing Systems · doi:10.48550/arxiv.1406.6909 · Cited by 1,080 · see citing works
Imitation Learning: A Survey of Learning Methods
Hussein, A., Gaber, M. M., Elyan, E., & Jayne, C. (2018) · ACM Computing Surveys · doi:10.1145/3054912 · Cited by 1,060 · see citing works
Newborn Infants Imitate Adult Facial Gestures
Meltzoff, A. N., & Moore, M. (1983) · Child Development · doi:10.1111/j.1467-8624.1983.tb00496.x · Cited by 956 · see citing works
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · see citing works
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762 · see citing works
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., & Abbeel, P. (2017) · arXiv · doi:10.48550/arxiv.1710.04615 · Cited by 618 · see citing works
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised LearningPreprint
Bardes, A., Ponce, J., & LeCun, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04906 · Cited by 617 · see citing works
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461249 · Cited by 609 · see citing works
Behavioral Cloning from Observation
Torabi, F., Warnell, G., & Stone, P. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/687 · Cited by 574 · see citing works
A survey of structure from motion.
Özyeşil, O., Voroninski, V., Basri, R., & Singer, A. (2017) · Acta Numerica · doi:10.1017/s096249291700006x · Cited by 501 · see citing works
Imitative learning of actions on objects by children, chimpanzees, and enculturated chimpanzees.
Tomasello, M., Savage-Rumbaugh, S., & Kruger, A. (1993) · Child Development · doi:10.1111/j.1467-8624.1993.tb04207.x · Cited by 462 · see citing works
Imitation of facial and manual gestures by human neonates
Meltzoff, A. N., & Moore, M. K. (1977) · Science · doi:10.1126/science.897687 · Cited by 460 · see citing works
One-Shot Imitation LearningPreprint
Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, O. J., Schneider, J., Sutskever, I., Abbeel, P., & Zaremba, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07326 · Cited by 449 · see citing works
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · arXiv · doi:10.48550/arxiv.2104.14548 · Cited by 405 · see citing works
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404 · see citing works
Imitative Learning of Actions on Objects by Children, Chimpanzees, and Enculturated Chimpanzees
Tomasello, M., Savage‐Rumbaugh, S., & Kruger, A. C. (1993) · Child Development · doi:10.2307/1131463 · Cited by 344 · see citing works
D4RL: Datasets for Deep Data-Driven Reinforcement LearningPreprint
Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07219 · Cited by 310 · see citing works
Reinforcement and Imitation Learning for Diverse Visuomotor Skills
Zhu, Y., Wang, Z., Merel, J., Rusu, A. A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R. T., de Freitas, N., et al. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.009 · Cited by 267 · see citing works
SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
Wang, Y., Chao, W.-L., Weinberger, K. Q., & van der Maaten, L. (2019) · arXiv · doi:10.48550/arxiv.1911.04623 · Cited by 245 · see citing works
Unsupervised Feature Learning via Non-Parametric Instance-level DiscriminationPreprint
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01978 · Cited by 189 · see citing works
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations
Song, S., Zeng, A., Lee, J., & Funkhouser, T. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3004787 · Cited by 185 · see citing works
Unsupervised Perceptual Rewards for Imitation LearningPreprint
Sermanet, P., Xu, K., & Levine, S. (2016) · Robotics: Science and Systems Conference · doi:10.48550/arxiv.1612.06699 · Cited by 170 · see citing works
Third-Person Imitation LearningPreprint
Stadie, B. C., Abbeel, P., & Sutskever, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01703 · Cited by 153 · see citing works
Florence, P., Lynch, C., Zeng, A., Ramirez, O., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., & Tompson, J. (2021) · arXiv · doi:10.48550/arxiv.2109.00137 · Cited by 131 · see citing works
Decoupling Representation Learning from Reinforcement Learning
Stooke, A., Lee, K., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2009.08319 · Cited by 123 · see citing works
Unsupervised Perceptual Rewards for Imitation Learning
Sermanet, P., Xu, K., & Levine, S. (2017) · Robotics: Science and Systems · doi:10.15607/rss.2017.xiii.050 · Cited by 117 · see citing works
Reinforcement and Imitation Learning for Diverse Visuomotor SkillsPreprint
Zhu, Y., Wang, Z., Merel, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09564 · Cited by 107 · see citing works
The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments
Kemp, C. C., Edsinger, A., Clever, H., & Matulevich, B. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811922 · Cited by 96 · see citing works
Towards Generalization and Simplicity in Continuous Control
Rajeswaran, A., Lowrey, K., Todorov, E., & Kakade, S. (2017) · arXiv · doi:10.48550/arxiv.1703.02660 · Cited by 94 · see citing works
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2107.09645 · Cited by 89 · see citing works
Self-citationReinforcement Learning with Prototypical RepresentationsPreprint
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11271 · Cited by 64 · see citing works
Self-citationKeypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement LearningPreprint
Manuelli, L., Li, Y., Florence, P., & Tedrake, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05085 · Cited by 55 · see citing works
Young, S., Gandhi, D., Tulsiani, S., Gupta, A., Abbeel, P., & Pinto, L. (2020) · arXiv · doi:10.48550/arxiv.2008.04899 · Cited by 42 · see citing works
Self-citationLearning to Navigate Using Mid-Level Visual PriorsPreprint
(Sasha), A. S., Zhang, J. O., Emi, B., Zamir, A., Savarese, S., Guibas, L. J., & Malik, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11121 · Cited by 33 · see citing works
DoorGym: A Scalable Door Opening Environment And Baseline Agent
Urakami, Y., Hodgkinson, A., Carlin, C., Leu, R., Rigazio, L., & Abbeel, P. (2019) · arXiv · doi:10.48550/arxiv.1908.01887 · Cited by 30 · see citing works
Learning to Catch: Applying Nearest Neighbor Algorithms to Dynamic Control Tasks
Aha, D. W., & Salzberg, S. L. (1994) · Lecture notes in statistics · doi:10.1007/978-1-4612-2660-4_33 · Cited by 28 · see citing works
RL Unplugged: Benchmarks for Offline Reinforcement Learning.Preprint
Gülçehre, Ç., Wang, Z., Novikov, A., Paine, T. L., Colmenarejo, S. G., Żołna, K., Agarwal, R., Merel, J., Mankowitz, D. J., Păduraru, C., et al. (2020) · arXiv (Cornell University) · Cited by 28 · see citing works
Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
Zhan, A., Zhao, R., Pinto, L., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2012.07975 · Cited by 28 · see citing works
Self-citationChen, B., (Sasha), A. S., Lewis, G., Armeni, I., Savarese, S., Zamir, A., Malik, J., & Pinto, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06698 · Cited by 23 · see citing works
Self-citationGrasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost DemonstrationsPreprint
Song, S., Zeng, A., Lee, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.04344 · Cited by 9 · see citing works
Simple Nearest Neighbor Policy Method for Continuous Control Tasks
Mansimov, E., & Cho, K. (2018) · Cited by 5 · see citing works
Playful Interactions for Representation Learning
Young, S., Pari, J., Abbeel, P., & Pinto, L. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981307 · Cited by 5 · see citing works
Self-citationRobust Reinforcement Learning with Relevance Vector Machines
Lee, M., & Anderson, C. W. (2016) · Cited by 3 · see citing works
The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human EnvironmentsPreprint
Kemp, C. C., Edsinger, A., Clever, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10892 · Cited by 2 · see citing works
Playful Interactions for Representation LearningPreprint
Young, S., Pari, J., Abbeel, P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09046
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [25]25. Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin'ichi Satoh. A survey of product quantization. ITE Transactions on Media Technology and Applications, 6 (1):2-10, 2018.
- [43]43. Faraz Torabi, Garrett Warnell, and Peter Stone. Be- havioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018. · arXiv
Cited by
Sorted by citations
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674 · see citing works
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
“Most works focus on pre-trained visual representations that can be used to initialize the encoder of the robot’s camera observations, either via supervised ImageNet classification [53], data augmentation [54, 55, 56, 57] or objectives that are tailored towards robotic control [58, 59, 60, 61, 62].”
BackgroundA Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182 · see citing works
Octo: An Open-Source Generalist Robot Policy
Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., Luo, J., et al. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.090 · Cited by 129 · see citing works
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119 · see citing works
“k-NN [82] finds trajectories in the dataset that are close (via distance in feature space [79]) to the goal image.”
BackgroundMethodAirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild
Fang, H., Fang, H.-S., Wang, Y., Ren, J., Chen, J., Zhang, R., Wang, W., & Lu, C. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610799 · Cited by 38 · see citing works
LEGATO: Cross-Embodiment Imitation Using a Grasping Tool
Seo, M., Park, H. A., Yuan, S., Zhu, Y., & Sentis, L. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3535182 · Cited by 11 · see citing works
On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation
Palo, N. D., & Johns, E. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3349832 · Cited by 11 · see citing works
“The use of retrieval for few-shot learning, especially in robotics, reinforcement learning and control, has also been investigated elsewhere [7], [12].”
BackgroundMethodInfluential
- Authors
- Pari, J., Shafiullah, N. M. (., Arunachalam, S. P., & Pinto, L.
- Affiliations
- New York University (US)
- Corresponding author
- Pinto, Lerrel
- Venue
- Robotics: Science and Systems XVIII
- Published
- 2022-06-27
- OpenAlex
- W3214952050
- Cited by
- 95
- References
- 62
- Citations by year
- 2026: 13 · 2025: 35 · 2024: 28 · 2023: 15 · 2022: 4
- FWCI
- 11.31
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Robotics: Science and Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- visual imitation learning, representation learning, self-supervised learning, locally weighted regression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
