Self-Supervised Correspondence in Visuomotor Policy Learning
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
In this letter, we explore using self-supervised correspondence for improving the generalization performance and sample efficiency of visuomotor policy learning. Prior work has primarily used approaches such as autoencoding, pose-based losses, and end-to-end policy optimization in order to train the visual portion of visuomotor policies. We instead propose an approach using self-supervised dense visual correspondence training and show that this enables visuomotor policy learning with surprisingly high generalization performance with modest amounts of data. Using imitation learning, we demonstrate extensive hardware validation on challenging manipulation tasks with as few as 50 demonstrations. Our learned policies can generalize across classes of objects, react to deformable object configurations, and manipulate textureless symmetrical objects in a variety of backgrounds, all with closed-loop, real-time vision-based policies. Simulated imitation learning experiments suggest that correspondence training offers sample complexity and generalization benefits compared to autoencoding and end-to-end training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · cited 1×
“While other works have shown that injecting noise into the dynamics either during imitation learning [33] or sim-to-real transfer [34] can alleviate cascading errors, we provide a simple method based only on data augmentation.”
BackgroundDynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298631 · Cited by 951 · cited 1×
“Other prior work [6] has used dynamic non-rigid reconstruction [35] to address dynamic scenes.”
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints
Mahjourian, R., Wicke, M., & Angelova, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00594 · Cited by 775
One-Shot Visual Imitation Learning via Meta-Learning
Finn, C., Yu, T., Zhang, T., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1709.04905 · Cited by 341 · cited 2×
“The model is close to [1]–[3] with the key architectural traits of having a few convolutional layers followed by a channel-wise spatial expectation operation, which has been widely used [10], [11], [25], [29], [36]–[38].”
BackgroundLearning Dexterous In-Hand Manipulation
OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., et al. (2018) · arXiv · doi:10.48550/arxiv.1808.00177 · Cited by 248
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
James, S., Davison, A. J., & Johns, E. (2017) · arXiv · doi:10.48550/arxiv.1707.02267 · Cited by 208 · cited 2×
“Another direction which has shown promising results is using sim-to-real transfer [14], [22]–[24], but our interest in a small amount of incremental human effort per new task is challenging for these methods, since they currently require significant engineering effort for each new simulation scenario.”
BackgroundDeep predictive policy training using reinforcement learning
Ghadirzadeh, A., Maki, A., Kragic, D., & Bjorkman, M. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206046 · Cited by 113
Research works printed but not linked: 29
- 10.1109/ICRA.2014.6907062 · DOI · Paper page
- 10.1109/ICCV.2017.623 · DOI
- laskey, 0, DART: Noise injection for robust imitation learning, Proc Conf Robot Learn, 143
- 10.15607/RSS.2019.XV.020 · DOI
- 10.1109/ICRA.2017.7989384 · DOI · Paper page
- finn, 0, Guided cost learning: Deep inverse optimal control via policy optimization, Proc Int Conf Mach Learn, 49
- manuelli, 0, kPAM: Keypoint affordances for category-level robotic manipulation, Proc Int Symp Robot Res
- 10.1109/IROS.2016.7759578 · DOI · Paper page
- 10.1109/ICRA.2018.8461076 · DOI · Paper page
- agrawal, 0, Learning to poke by poking: Experiential learning of intuitive physics, Proc Adv Neural Inf Process Syst, 5074
- ebert, 2018, Visual foresight: Model-based deep reinforcement learning for vision-based robotic control, arXiv 1812 00568
- 10.1109/ICRA.2018.8462891 · DOI · Paper page
- 10.15607/RSS.2017.XIII.050 · DOI · Paper page
- kulkarni, 0, Unsupervised learning of object keypoints for perception and control, Proc Adv Neural Inf Process Syst
- 10.1007/978-3-540-30301-5_60 · DOI · Paper page
- 10.1109/ICRA.2018.8461249 · DOI · Paper page
- 10.1109/LRA.2016.2634089 · DOI · Paper page
- 10.1109/IROS.2017.8202141 · DOI · Paper page
- 10.1109/ICRA.2019.8793621 · DOI · Paper page
- kroemer, 2019, A review of robot learning for manipulation: Challenges, representations, and algorithms, arXiv 1907 03146
- florence, 0, Dense object nets: Learning dense visual object descriptors by and for robotic manipulation, Proc Conf Robot Learn, 373
- 10.1109/ICRA.2016.7487173 · DOI · Paper page
- 10.1109/LRA.2016.2633383 · DOI · Paper page
- levine, 2016, End-to-end training of deep visuomotor policies, J Mach Learn Res, 17, 1
- sieb, 0, Graph-structured visual imitation, Proc Conf Robot Learn
- 10.15607/RSS.2018.XIV.009 · DOI · Paper page
- 10.1109/ICRA.2016.7487517 · DOI · Paper page
- matas, 0, Sim-to-real reinforcement learning for deformable object manipulation, Proc Conf Robot Learn, 734
- 10.15607/RSS.2018.XIV.002 · DOI · Paper page
Cited by
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2304.13705 · Cited by 673
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674
Diffusion policy: Visuomotor policy learning via action diffusion
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., & Song, S. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241273668 · Cited by 613
“The simplest form of explicit policies maps from world state or observation directly to action (Pomerleau, 1988; Zhang et al., 2018; Florence et al., 2020; Ross et al., 2011; Toyer et al., 2020; Rahmatizadeh et al., 2018); Bojarski et al., 2016).”
InfluentialSelf-citationHow to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
“…methods such as autoencoders (Finn et al. 2016b; Ghadirzadeh et al. 2017), contrastive losses (Sermanet et al. 2018), and correspondence learning (Florence et al. 2018, 2019), provide a reasonable solution in cases where the inductive biases of the unsupervised algorithm effectively match the…”
MethodDiffusion Policy: Visuomotor Policy Learning via Action Diffusion
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., & Song, S. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.026 · Cited by 530
Metadata
- Authors
- Florence, P., Manuelli, L., & Tedrake, R.
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2020-04
- Volume
- 5
- Issue
- 2
- Pages
- 492-499
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2972997256
- Cited by
- 110
- References
- 10
- FWCI
- 5.30
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Science Foundation (IIS-1427050)
- Lockheed Martin Corporation (RPP2016-002)
- Amazon Research Award Grant
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Self-Supervised Correspondence in Visuomotor Policy Learning. https://app.cochat.ai/papers/10.1109/lra.2019.2956365
- The paper
- Florence, P., Manuelli, L., & Tedrake, R. (2020). Self-Supervised Correspondence in Visuomotor Policy Learning. IEEE Robotics and Automation Letters. https://doi.org/10.1109/lra.2019.2956365
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.