The DOI matches its DataCite record; 63 of the 64 references it cites were found in a registry or an index.
CLIPort: What and Where Pathways for Robotic Manipulation
Open accessAbstract
How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While CLIPort (Shridhar et al., 2021) and MOO (Stone et al., 2023) integrate pre-trained VLMs into end-to-end visuomotor manipulation policies, both incorporate significant structure into the policy that limits their applicability .”
From RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023) · cites this paper as Shridhar et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007
Segregation of Form, Color, Movement, and Depth: Anatomy, Physiology, and Perception
Livingstone, M. S., & Hubel, D. H. (1988) · Science · doi:10.1126/science.3283936 · Cited by 3,358
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A. J., & Zisserman, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.213 · Cited by 2,842
RECEPTIVE FIELDS AND FUNCTIONAL ARCHITECTURE IN TWO NONSTRIATE VISUAL AREAS (18 AND 19) OF THE CAT
Hubel, D. H., & Wiesel, T. N. (1965) · Journal of Neurophysiology · doi:10.1152/jn.1965.28.2.229 · Cited by 2,615
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.019 · Cited by 2,139
Self-citationViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
A simple framework for contrastive learning of visual representations
Chen, T. (2024) · TIB Data Manager · doi:10.57702/iuuvgtaz · Cited by 1,140
Group Equivariant Convolutional NetworksPreprint
Cohen, T., & Welling, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07576 · Cited by 1,106
Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque.
Derrington, A. M., & Lennie, P. (1984) · The Journal of Physiology · doi:10.1113/jphysiol.1984.sp015498 · Cited by 1,059
Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation
Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A., Teller, S. J., & Roy, N. (2011) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7979 · Cited by 903
The Ecological Approach to Visual Perception: Classic Edition
Gibson, J. J. (2014) · Cited by 896
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 802
Kondor, R., & Trivedi, S. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.03690 · Cited by 684
Solving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602
Zeng, A., Song, S., Yu, K., Donlon, E., Hogan, F. R., Bauzá, M., Ma, D., Taylor, O., Liu, M., Romo, E., et al. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461044 · Cited by 498
Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge
Zeng, A., Yu, K., Song, S., Suo, D., Walker, E., Rodríguez, A., & Xiao, J. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989165 · Cited by 456
(2021) · International Conference on Learning Representations · Cited by 449
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019) · arXiv · doi:10.48550/arxiv.1904.12584 · Cited by 444
12-in-1: Multi-Task Vision and Language Representation Learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01045 · Cited by 420
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at ScalePreprint
Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jonschkowski, R., Finn, C., Levine, S., & Hausman, K. (2021) · arXiv.org · doi:10.48550/arxiv.2104.08212 · Cited by 344
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 330
Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions
Misra, D., Sung, J., Lee, K., & Saxena, A. (2014) · Int. J. Robotics Res. · doi:10.1177/0278364915602060 · Cited by 313
Audiovisual SlowFast Networks for Video RecognitionPreprint
Xiao, F., Lee, Y. J., Grauman, K. L., Malik, J., & Feichtenhofer, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08740 · Cited by 251
Self-supervised 6D Object Pose Estimation for Robot Manipulation
Deng, X., Yu, X., Mousavian, A., Eppner, C., Bretl, T. W., & Fox, D. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196714 · Cited by 219
Self-citationSingle image 3D object detection and pose estimation for grasping
Zhu, M., Derpanis, K. G., Yang, Y., Brahmbhatt, S., Zhang, M. M., Phillips, C. J., Lecce, M., & Daniilidis, K. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907430 · Cited by 217
Transporter Networks: Rearranging the Visual World for Robotic Manipulation
Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., Wahid, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14406 · Cited by 200
KPAM: KeyPoint Affordances for Category-Level Robotic Manipulation
Manuelli, L., Gao, W., Florence, P., & Tedrake, R. (2022) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-95459-8_9 · Cited by 198
Self-citationMapping Instructions to Actions in 3D Environments with Visual Goal Prediction
Misra, D. K., Bennett, A., Blukis, V., Niklasson, E., Shatkhin, M., & Artzi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1287 · Cited by 183
Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic ManipulationPreprint
Florence, P., Manuelli, L., & Tedrake, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08756 · Cited by 181
Self-citationLearning to Manipulate Deformable Objects without Demonstrations
Wu, Y., Yan, W., Kurutach, T., Pinto, L., & Abbeel, P. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.065 · Cited by 181
Unseen Object Instance Segmentation for Robotic Environments
Xie, C., Yu, X., Mousavian, A., & Fox, D. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3060341 · Cited by 175
Self-citationInteractively Picking Real-World Objects with Unconstrained Spoken Language Instructions
Hatori, J., Kikuchi, Y., Kobayashi, S., Takahashi, K., Tsuboi, Y., Unno, Y., Ko, W., & Tan, J. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460699 · Cited by 174
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
Liu, X., Jonschkowski, R., Angelova, A., & Konolige, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01162 · Cited by 140
Learning to interpret natural language commands through human-robot dialog
Thomason, J., Zhang, S., Mooney, R. J., & Stone, P. (2015) · Cited by 138
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction
Shridhar, M., & Hsu, D. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.028 · Cited by 132
Self-citationSeita, D., Florence, P., Tompson, J., Coumans, E., Sindhwani, V., Goldberg, K., & Zeng, A. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561391 · Cited by 130
Unsupervised Learning of Object Keypoints for Perception and ControlPreprint
Kulkarni, T. D., Gupta, A., Ionescu, C., Borgeaud, S., Reynolds, M., Zisserman, A., & Mnih, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.11883 · Cited by 128
Self-Supervised Correspondence in Visuomotor Policy Learning
Florence, P., Manuelli, L., & Tedrake, R. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2956365 · Cited by 111
Self-citationForm2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly
Zakka, K., Zeng, A., Lee, J., & Song, S. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196733 · Cited by 109
Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data
Sundaresan, P., Grannen, J., Thananjeyan, B., Balakrishna, A., Laskey, M., Stone, K., Gonzalez, J. E., & Goldberg, K. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197121 · Cited by 107
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene GraphPreprint
Yu, F. R., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16934 · Cited by 106
Natural Language Communication with Robots
Bisk, Y., Yüret, D., & Marcu, D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1089 · Cited by 103
Learning from Unscripted Deictic Gesture and Language for Human-Robot Interactions
Matuszek, C., Bo, L., Zettlemoyer, L., & Fox, D. (2014) · Maryland Shared Open Access Repository (USMAI Consortium) · doi:10.13016/m2rn30b52 · Cited by 81
Self-citationEnd-to-End Learning of Semantic GraspingPreprint
Jang, E., Vijayanarasimhan, S., Pástor, P., Ibarz, J., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01932 · Cited by 75
Learning to See before Learning to Act: Visual Pre-training for Manipulation
Lin, Y.-C., Zeng, A., Song, S., Isola, P., & Lin, T.-Y. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197331 · Cited by 66
Interpreting and Executing Recipes with a Cooking Robot.
Bollini, M., Tellex, S., Thompson, T., Roy, N., & Rus, D. L. (2012) · Cited by 55
A Joint Network for Grasp Detection Conditioned on Natural Language Commands
Chen, Y., Xu, R., Lin, Y., & Vela, P. A. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561994 · Cited by 50
Prospection: Interpretable plans from language by predicting the future
Paxton, C., Bisk, Y., Thomason, J., Byravan, A., & Fox, D. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794441 · Cited by 39
Self-citationGrounding Language in Play.Preprint
Lynch, C., & Sermanet, P. (2020) · arXiv (Cornell University) · Cited by 35
Ding, X. D., Hill, F., Santoro, A., & Botvinick, M. (2020) · arXiv (Cornell University) · Cited by 22
Research works printed but not linked: 7
- [4][4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for visionbased robotic manipulation. Conference on Robot Learning (CoRL) , 2018. -
- [21][21] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick. Mask r-cnn. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. -
- [26][26] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL) , pages 1369-1378. PMLR, 2020. -
- [30][30] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for categorylevel robotic manipulation. In International Symposium on Robotics Research (ISRR) , 2019. -
- [34][34] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closedloop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters , 5(3): 4978-4985, 2020. -
- [45][45] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding for mapping natural language instructions to robot control. In Conference on Robot Learning (CoRL) , 2020. -
- [52][52] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Slow-fast auditory streams for audio recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 855-859. IEEE, 2021. -
Cited by
Sorted by citations
(2022) · doi:10.15607/rss.2023.xix.025 · Cited by 916
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2304.13705 · Cited by 673
End-to-End Autonomous Driving: Challenges and Frontiers
Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., & Li, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3435937 · Cited by 843
Metadata
- Authors
- Shridhar, M., Manuelli, L., & Fox, D.
- Affiliations
- University of Washington (US)
- Nvidia (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2109.12098
- OpenAlex
- W3203511201
- Cited by
- 223
- References
- 64
- Citations by year
- 2026: 2 · 2025: 31 · 2024: 107 · 2023: 65 · 2022: 16 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Conference on Robot Learning
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- robotic manipulation, vision-based manipulation, CLIP, transporters, dual-stream architecture, spatial reasoning, semantic representation, cloth folding, few-shot generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
