The DOI matches its DataCite record; 63 of the 64 references it cites were found in a registry or an index.
CLIPort: What and Where Pathways for Robotic Manipulation
Open accessAbstract
How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While CLIPort (Shridhar et al., 2021) and MOO (Stone et al., 2023) integrate pre-trained VLMs into end-to-end visuomotor manipulation policies, both incorporate significant structure into the policy that limits their applicability .”
From RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023) · cites this paper as Shridhar et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Research works printed but not linked: 7
- [4][4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for visionbased robotic manipulation. Conference on Robot Learning (CoRL) , 2018. -
- [21][21] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick. Mask r-cnn. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017. -
- [26][26] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on Robot Learning (CoRL) , pages 1369-1378. PMLR, 2020. -
- [30][30] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for categorylevel robotic manipulation. In International Symposium on Robotics Research (ISRR) , 2019. -
- [34][34] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closedloop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters , 5(3): 4978-4985, 2020. -
- [45][45] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding for mapping natural language instructions to robot control. In Conference on Robot Learning (CoRL) , 2020. -
- [52][52] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Slow-fast auditory streams for audio recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 855-859. IEEE, 2021. -
Cited by
Sorted by citations
(2022) · doi:10.15607/rss.2023.xix.025 · Cited by 916
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2304.13705 · Cited by 673
End-to-End Autonomous Driving: Challenges and Frontiers
Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., & Li, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3435937 · Cited by 843
Image Segmentation Using Text and Image Prompts
Lüddecke, T., & Ecker, A. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00695 · Cited by 657
ChatGPT for Robotics: Design Principles and Model Abilities
Vemprala, S. H., Bonatti, R., Bucker, A., & Kapoor, A. (2024) · IEEE Access · doi:10.1109/access.2024.3387941 · Cited by 570
Visual Language Maps for Robot Navigation
Huang, C., Mees, O., Zeng, A., & Burgard, W. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160969 · Cited by 376
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287
“CLIP has been employed in several robotics and embodied settings in zero-shot manner [74], or combined with Transporter networks [75] as in CLIPort [76].”
BackgroundMethodSpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01370 · Cited by 281
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
“This simple approach is in contrast with prior alternatives for incorporating VLMs into robot policies [21] or designing new vision-language-action architectures from scratch [22]: instead, pre-existing vision-language models, with already-amortized significant compute investment, are trained…”
MethodSpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F. (2024) · arXiv · doi:10.48550/arxiv.2401.12168 · Cited by 269
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610243 · Cited by 406
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 267
Text2Motion: from natural language instructions to feasible plans
Lin, K., Agia, C., Migimatsu, T., Pavone, M., & Bohg, J. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10131-7 · Cited by 343
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration 0
O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611477 · Cited by 210
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222
“[20] Mohit Shridhar, Lucas Manuelli, and Dieter Fox.”
BackgroundMees, O., Hermann, L., Rosete-Beas, E., & Burgard, W. B. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3180108 · Cited by 189
Simple but Effective: CLIP Embeddings for Embodied AI
Khandelwal, A., Weihs, L., Mottaghi, R., & Kembhavi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01441 · Cited by 172
Embodied large language models enable robots to complete complex tasks in unpredictable environments
Mon-Williams, R., Li, G., Long, R., Du, W., & Lucas, C. G. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01005-x · Cited by 132
Affordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119
“grasping, pushing or placing) to each location [105, 124, 125].”
BackgroundCLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Shafiullah, N. M. M., Paxton, C., Pinto, L., Chintala, S., & Szlam, A. D. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.074 · Cited by 172
Semantic anomaly detection with large language models
Elhafsi, A., Sinha, R., Agia, C., Schmerling, E., Nesnas, I. A. D., & Pavone, M. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10132-6 · Cited by 101
DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
Kapelyukh, I., Vosylius, V., & Johns, E. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3272516 · Cited by 114
@ CREPE: Can Vision-Language Foundation Models Reason Compositionally?
Ma, Z., Hong, J., Gul, M. O., Gandhi, M., Gao, I., & Krishna, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01050 · Cited by 111
Bharadhwaj, H., Vakil, J., Sharma, M. K., Gupta, A. K., Tulsiani, S., & Kumar, V. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611293 · Cited by 138
Correcting Robot Plans with Natural Language Feedback
Sharma, P., Sundaralingam, B., Blukis, V., Paxton, C., Hermans, T., Torralba, A. B., Andreas, J., & Fox, D. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.065 · Cited by 138
Self-citationScaling Robot Learning with Semantically Imagined Experience
Yu, T., Xiao, T. G., Tompson, J., Stone, A., Wang, S., Brohan, A., Singh, J., Tan, C., Dee, M., Peralta, J., et al. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.027 · Cited by 226
What matters in language conditioned robotic imitation learning over unstructured data
Mees, O., Hermann, L., & Burgard, W. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3196123 · Cited by 77
Language-Driven Representation Learning for Robotics
Karamcheti, S., Nair, S., Chen, A., Kollar, T., Finn, C., Sadigh, D., & Liang, P. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.032 · Cited by 185
Grounding Language with Visual Affordances over Unstructured Data
Mees, O., Diaz, J. L. B., & Burgard, W. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160396 · Cited by 100
RT-H: Action Hierarchies Using Language
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., & Sadigh, D. (2024) · arXiv · doi:10.48550/arxiv.2403.01823 · Cited by 71
CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models
Huang, H.-X., Lin, F., Hu, Y.-D., Wang, S.-J., & Gao, Y. (2024) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros58592.2024.10801352 · Cited by 182
A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter
Xu, K., Zhao, S., Zhou, Z., Li, Z., Pi, H., Zhu, Y., Wang, Y., & Xiong, R. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161041 · Cited by 72
OpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62
“Towards this goal, existing work has explored integrating pretrained language and vision-language models for robotic representation learning [12–14] and as a component in modular systems for task planning and execution [15, 16].”
BackgroundChat with the Environment: Interactive Multimodal Perception Using Large Language Models
Zhao, X., Li, M., Weber, C., Hafez, M. B., & Wermter, S. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342363 · Cited by 62
Fan, J., & Zheng, P. (2024) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2024.05.003 · Cited by 64
LEGO-Net: Learning Regular Rearrangements of Objects in Rooms
Wei, Q. A., Ding, S., Park, J. J., Sajnani, R., Poulenard, A., Sridhar, S., & Guibas, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01825 · Cited by 60
Cui, Y., Karamcheti, S., Palleti, R., Shivakumar, N., Liang, P., & Sadigh, D. (2023) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3568162.3578623 · Cited by 85
LATTE: LAnguage Trajectory TransformEr
Bucker, A., Figueredo, L., Haddadin, S., Kapoor, A., Ma, S., Vemprala, S. H., & Bonatti, R. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161068 · Cited by 70
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
Yang, Y., Jia, B., Zhi, P., & Huang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01539 · Cited by 83
Bucker, A., Figueredo, L., Haddadinl, S., Kapoor, A., Ma, S., & Bonatti, R. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981810 · Cited by 55
Task-Oriented Grasp Prediction with Visual-Language Inputs
Tang, C., Huang, D., Meng, L., Liu, W., & Zhang, H. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342268 · Cited by 53
GaussianGrasper: 3D Language Gaussian Splatting for Open-Vocabulary Robotic Grasping
Zheng, Y., Chen, X., Zheng, Y., Gu, S., Yang, R., Jin, B., Li, P., Zhong, C., Wang, Z., Liu, L., et al. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3432348 · Cited by 49
Robot learning in the era of foundation models: a survey
Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Jiang, S., He, B., & Cheng, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129963 · Cited by 56
Structured World Models from Human Videos
Mendonca, R., Bahl, S., & Pathak, D. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.012 · Cited by 107
RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
Vecerík, M., Doersch, C., Yang, Y., Davchev, T., Aytar, Y., Zhou, G., Hadsell, R. T., Agapito, L., & Scholz, J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611409 · Cited by 44
Learning Navigational Visual Representations with Semantic Map Supervision
Hong, Y., Zhou, Y., Zhang, R., Dernoncourt, F., Bui, T., Gould, S., & Tan, H. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00284 · Cited by 42
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
Xiao, T. G., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., & Tompson, J. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.029 · Cited by 84
Statler: State-Maintaining Language Models for Embodied Reasoning
Yoneda, T., Fang, J., Li, P., Zhang, H., Jiang, T., Lin, S., Picker, B., Yunis, D., Mei, H., & Walter, M. R. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610634 · Cited by 62
Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning
Sun, J., Zhang, Q., Duan, Y., Jiang, X., Chen, C., & Xu, R. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610948 · Cited by 52
CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language
Sanghi, A., Fu, R., Liu, V., Willis, K. D. D., Shayani, H., Khasahmadi, A. H., Sridhar, S., & Ritchie, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01759 · Cited by 49
Language-driven Grasp Detection
Vuong, A. D., Vu, M. N., Huang, B., Nguyen, N., Le, H., Vo, T., & Nguyen, A. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01695 · Cited by 39
Embodied Multi-Agent Systems: A Review
Li, Z., Wu, W., Guo, Y., Sun, J. Q., & Han, Q. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125552 · Cited by 42
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment
Guo, Y., Wang, Y.-J., Zha, L.-H., Jiang, Z., & Chen, J. (2023) · IEEE/RJS International Conference on Intelligent RObots and Systems · doi:10.1109/iros58592.2024.10802284 · Cited by 118
StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
Liu, W., Du, Y., Hermans, T., Chernova, S., & Paxton, C. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.031 · Cited by 72
Wu, D., Zheng, P., Zhao, Q., Zhang, S., Qi, J., Hu, J., Zhu, G., & Wang, L. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103064 · Cited by 38
Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement
Goodwin, W., Vaze, S., Havoutis, I., & Posner, I. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811817 · Cited by 35
IFOR: Iterative Flow Minimization for Robotic Object Rearrangement
Goyal, A., Mousavian, A., Paxton, C., Chao, Y.-W., Okorn, B., Deng, J., & Fox, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01437 · Cited by 35
Self-citationA Simple Long-Tailed Recognition Baseline via Vision-Language ModelPreprint
Ma, T., Geng, S., Wang, M., Shao, J., Lu, J., Li, H., Gao, P., & Qiao, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14745 · Cited by 57
CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents
Park, J., Lim, S., Lee, J., Park, S., Chang, M., Yu, Y., & Choi, S. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3338514 · Cited by 44
GRID: Scene-Graph-based Instruction-driven Robotic Task Planning
Ni, Z., Deng, X., Tai, C., Zhu, X., Xie, Q., Huang, W., Wu, X., & Zeng, L. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801291 · Cited by 42
Exploring Embodied Multimodal Large Models: Development, datasets, and future directions
Chen, S., Wu, Z., Zhang, K., Li, C., Zhang, B., Ma, F., Yu, F. R., & Li, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103198 · Cited by 39
Gong, R., Huang, J., Zhao, Y.-Z., Geng, H., Gao, X.-F., Wu, Q., Ai, W., Zhou, Z.-H., Terzopoulos, D., Zhu, S.-C., et al. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01873 · Cited by 96
DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models
Palo, N. D., & Johns, E. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610923 · Cited by 30
Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations
Gbagbe, K. F., Cabrera, M. A., Alabbas, A., Alyunes, O., Lykov, A., & Tsetserukou, D. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831380 · Cited by 29
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., & Fox, D. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.133 · Cited by 29
Self-citationDecomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
Xie, A., Lee, L., Xiao, T., & Finn, C. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611331 · Cited by 28
Sivaprakasam, M., Maheshwari, P., Castro, M. G., Triest, S., Nye, M., Willits, S. M., Saba, A., Wang, W., & Scherer, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611265 · Cited by 28
SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphs
Zhai, G., Cai, X., Huang, D., Di, Y., Manhardt, F., Tombari, F., Navab, N., & Busam, B. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610792 · Cited by 33
Özalp, R., Uçar, A., & Guzelis, C. (2024) · IEEE Access · doi:10.1109/access.2024.3385426 · Cited by 27
Aligning Human and Robot Representations
Bobu, A., Peng, A., Agrawal, P., Shah, J., & Dragan, A. D. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610977.3634987 · Cited by 55
Huang, S., Ponomarenko, I., Jiang, Z., Li, X., Hu, X., Gao, P., Li, H., & Dong, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801993 · Cited by 43
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24
“Planar actions [10, 37] aren’t flexible enough to manipulate prismatic and revolute joints.”
BackgroundInfluentialApplication of Pretrained Large Language Models in Embodied Artificial Intelligence
Kovalev, A. K., & Panov, A. I. (2022) · Doklady Mathematics · doi:10.1134/s1064562422060138 · Cited by 23
Skill Transformer: A Monolithic Policy for Mobile Manipulation
Huang, X., Batra, D., Rai, A., & Szot, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00996 · Cited by 23
Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets
Du, M., Nair, S., Sadigh, D., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.011 · Cited by 20
Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flight
Xing, J., Bauersfeld, L., Song, Y., Xing, C., & Scaramuzza, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610095 · Cited by 22
Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models
Kapelyukh, I., Ren, Y., Alzugaray, I., & Johns, E. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611220 · Cited by 21
Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors
Kawaharazuka, K., Obinata, Y., Kanazawa, N., Okada, K., & Inaba, M. (2023) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids57100.2023.10375211 · Cited by 21
Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement
Gkanatsios, N., Jain, A., Zhou, X., Zhang, Y., Atkeson, C. G., & Fragkiadaki, K. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.030 · Cited by 35
Text2Reaction : Enabling Reactive Task Planning Using Large Language Models
Yang, Z., Li, N., Wang, H., Jiang, T., Zhang, S., Cui, S., Jiang, H., Li, C., Wang, S., & Wang, Z. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3371223 · Cited by 21
Lifelong Robot Learning with Human Assisted Language Planners
Parakh, M., Fong, A., Simeonov, A., Chen, T., Gupta, A., & Agrawal, P. (2023) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57147.2024.10610225 · Cited by 28
RT-Grasp: Reasoning Tuning Robotic Grasping via Multi-modal Large Language Model
Xu, J., Jin, S., Lei, Y., Zhang, Y., & Zhang, L. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801718 · Cited by 21
Liu, Y., Liu, S., Chen, B., Yang, Z.-X., & Xu, S. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3539193 · Cited by 20
A survey of Semantic Reasoning frameworks for robotic systems
Liu, W., Daruna, A., Patel, M., Ramachandruni, K., & Chernova, S. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104294 · Cited by 20
Seeing the Unseen: Visual Common Sense for Semantic Placement
Ramrakhya, R., Kembhavi, A., Batra, D., Kira, Z., Zeng, K.-H., & Weihs, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01540 · Cited by 19
Learning Video-Conditioned Policies for Unseen Manipulation Tasks
Chane-Sane, E., Schmid, C., & Laptev, I. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161336 · Cited by 17
Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimization
Lew, T., Singh, S., Prats, M., Bingham, J., Weisz, J., Holson, B., Zhang, X., Sindhwani, V., Lu, Y., Xia, F., et al. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161283 · Cited by 17
Learning Language-Conditioned Deformable Object Manipulation with Graph Dynamics
Deng, Y., Mo, K., Xia, C., & Wang, X. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610890 · Cited by 17
ReorientDiff: Diffusion Model based Reorientation for Object Manipulation
Mishra, U. A., & Chen, Y. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610749 · Cited by 25
Language-Conditioned Imitation Learning With Base Skill Priors Under Unstructured Data
Zhou, H., Bing, Z., Yao, X., Su, X., Yang, C., Huang, K., & Knoll, A. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3466076 · Cited by 17
Leveraging large language models for comprehensive locomotion control in humanoid robots design
Sun, S., Li, C., Zhao, Z., Huang, H., & Xu, W. (2024) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2024.100187 · Cited by 18
Transferring Foundation Models for Generalizable Robotic Manipulation
Yang, J., Tan, W., Jin, C., Yao, K., Liu, B., Fu, J., Song, R., Wu, G., & Wang, L. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00201 · Cited by 35
Generalizable Task Planning Through Representation Pretraining
Wang, C., Xu, D., & Fei-Fei, L. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3186635 · Cited by 17
Learning Neuro-symbolic Programs for Language Guided Robot Manipulation
Namasivayam, K., Singh, H., Bindal, V., Tuli, A., Agrawal, V., Jain, R., Singla, P., & Paul, R. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160545 · Cited by 17
Liu, X., Huang, C., Zhu, H., Wang, Z., Li, J., & Cangelosi, A. (2024) · IEEE Systems Man and Cybernetics Magazine · doi:10.1109/msmc.2023.3238855 · Cited by 16
Unlocking Robotic Autonomy: A Survey on the Applications of Foundation Models
Jang, D.-S., Cho, D.-H., Lee, W.-C., Ryu, S.-K., Jeong, B.-M., Hong, M., Jung, M., Kim, M., Lee, M., Lee, S., et al. (2024) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0438-7 · Cited by 17
CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation
Wang, J., Qin, Y., Kuang, K., Korkmaz, Y., Gurumoorthy, A., Su, H., & Wang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01700 · Cited by 21
Metadata
- Authors
- Shridhar, M., Manuelli, L., & Fox, D.
- Affiliations
- University of Washington (US)
- Nvidia (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2109.12098
- OpenAlex
- W3203511201
- Cited by
- 223
- References
- 64
- Citations by year
- 2026: 2 · 2025: 31 · 2024: 107 · 2023: 65 · 2022: 16 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Conference on Robot Learning
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- robotic manipulation, vision-based manipulation, CLIP, transporters, dual-stream architecture, spatial reasoning, semantic representation, cloth folding, few-shot generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
