ManipulaTHOR: A Framework for Visual Object Manipulation
Open accessAbstract
The domain of Embodied AI has recently witnessed substantial progress, particularly in navigating agents within their environments. These early successes have laid the building blocks for the community to tackle tasks that require agents to actively interact with objects in their environment. Object manipulation is an established research domain within the robotics community and poses several challenges including manipulator motion, grasping and long-horizon planning, particularly when dealing with oft-overlooked practical setups involving visually rich and complex scenes, manipulation using mobile agents (as opposed to tabletop manipulation), and generalization to unseen environments and objects. We propose a framework for object manipulation built upon the physics-enabled, visually rich AI2-THOR framework and present a new challenge to the Embodied AI community known as ArmPointNav. This task extends the popular point navigation task [2] to object manipulation and offers new challenges including 3D obstacle avoidance, manipulating objects in the presence of occlusion, and multi-object manipulation that necessitates long term planning. Popular learning paradigms that are successful on PointNav challenges show promise, but leave a large room for improvement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing vision-and-language navigation datasets face a common data scarcity issue.
“To tackle the common data scarcity issue in existing vision-and-language navigation datasets”
From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [24]
The Introduction mentions navigation and instruction following, while the Experiments section describes an object-manipulation dataset. Neither addresses whether existing vision-and-language navigation datasets have scarce data. Searches for data scarcity and limited training data in language-guided navigation found no passage making that point. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
AI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578 · see citing works
Self-citationSAPIEN: A SimulAted Part-Based Interactive ENvironment
Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01111 · Cited by 406 · see citing works
Rearrangement: A Challenge for Embodied AI
Batra, D., Chang, A. X., Chernova, S., Davison, A. J., Deng, J., Koltun, V., Levine, S., Malik, J., Mordatch, I., Mottaghi, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.01975 · Cited by 109 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 39
- Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. arXiv, 2018. 1, 2, 4
- Antonio Bicchi and Vijay Kumar. Robotic grasping and con- tact: A review. In ICRA, 2000. 2
- Jeannette Bohg, Antonio Morales, Tamim Asfour, and Dan- ica Kragic. Data-driven grasp synthesis-a survey. IEEE Transactions on Robotics, 2013. 2
- Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Vanhoucke. Using simulation and domain adapta- tion to improve efficiency of deep robotic grasping. In ICRA, 2018. 2
- Devendra Singh Chaplot, Saurabh Gupta, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to ex- plore using active neural mapping. In ICLR, 2020. 3
- Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. 6
- Kuan Fang, Yunfei Bai, Stefan Hinterstoisser, Silvio Savarese, and Mrinal Kalakrishnan. Multi-task domain adap- tation for deep learning of instance grasping from simulation. In ICRA, 2018. 2
- Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning task- oriented grasping for tool manipulation from simulated self- supervision. The International Journal of Robotics Research, 2020. 2
- Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Dynamics learning with cascaded variational in- ference for multi-step manipulation. In CoRL, 2020. 2
- Ronald Fearing. Implementing a force strategy for object re-orientation. In ICRA, 1986. 2
- Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threed- world: A platform for interactive multi-modal physical sim- ulation. arXiv, 2020. 2
- Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, 2020. 5
- Caelan Reed Garrett, Chris Paxton, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Dieter Fox. Online replanning in belief space for partially observable task and motion prob- lems. In ICRA, 2020. 2
- Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashc- hand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. In NeurIPS, 2018. 3
- Saurabh Gupta, James Davidson, Sergey Levine, Rahul Suk- thankar, and Jitendra Malik. Cognitive mapping and plan- ning for visual navigation. In CVPR, 2017. 3
- T. Haarnoja, Vitchyr H. Pong, Aurick Zhou, Murtaza Dalal, P. Abbeel, and S. Levine. Composable deep reinforcement learning for robotic manipulation. In ICRA, 2018. 2
- Stephen James, Zicong Ma, David Rovick Arrojo, and An- drew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Let- ters, 2020. 3
- Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data col- lection. The International Journal of Robotics Research, 2018. 2
- Chengshu Li, Fei Xia, Roberto Martín-Martín, and Silvio Savarese. Hrl4in: Hierarchical reinforcement learning for in- teractive navigation with mobile manipulators. arXiv, 2019. 2
- Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Em- mons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowd- sourcing platform for robotic skill learning through imita- tion. arXiv, 2018. 3
- Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Ku- maran, and Raia Hadsell. Learning to navigate in complex environments. In ICLR, 2017. 3
- Adithyavairavan Murali, Arsalan Mousavian, Clemens Epp- ner, Chris Paxton, and Dieter Fox. 6-dof grasping for target- driven object manipulation in clutter. In ICRA, 2020. 2, 5
- Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In ICLR, 2018. 3
- Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A plat- form for embodied ai research. In ICCV, 2019. 3, 8
- Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín- Martín, Linxi Fan, Guanzhi Wang, Shyamal Buch, Claudia D'Arpino, Sanjana Srivastava, Lyne P Tchapmi, et al. igib- son, a simulation environment for interactive tasks in large realisticscenes. arXiv, 2020. 3
- Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020. 5
- Siddharth Srivastava, Eugene Fang, Lorenzo Riano, Ro- han Chitnis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible planner- independent interface layer. In ICRA, 2014. 5
- Mike Stilman, Jan-Ullrich Schamburek, James Kuffner, and Tamim Asfour. Manipulation planning among movable ob- stacles. In ICRA, 2007. 2
- Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo- Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: A framework for embodied ai research. arXiv, 2020. 6
- Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2020. 1, 3, 5, 6
- Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In CVPR, 2019. 3
- Fei Xia, Chengshu Li, Roberto Martín-Martín, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Leverag- ing motion generation in reinforcement learning for mobile manipulation. arXiv, 2020. 2
- Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Mi- cael Edmond Tchapmi, Alexander Toshev, Roberto Martín- Martín, and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environ- ments. IEEE Robotics and Automation Letters, 2020. 2, 3
- Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene representations for robot manip- ulation. In CoRL, 2020. 2
- Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebo- tar, and Sergey Levine. Collective robot reinforcement learn- ing with distributed asynchronous guided policy search. In IROS, 2017. 2
- Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. In ICLR, 2019. 3
- Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020. 2
- Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei- Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Vi- sual semantic planning using deep successor representations. In ICCV, 2017. 5
- Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017. 3
Cited by
Sorted by citations
Simple but Effective: CLIP Embeddings for Embodied AI
Khandelwal, A., Weihs, L., Mottaghi, R., & Kembhavi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01441 · Cited by 172 · see citing works
Self-citationScaling Data Generation in Vision-and-Language Navigation
Wang, Z., Li, J., Hong, Y., Wang, Y., Wu, Q., Bansal, M., Gould, S., Tan, H., & Qiao, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01103 · Cited by 64 · see citing works
“Large-Scale Visual Navigation Learning Due to the expensive navigational data collection process, learning to navigate usually faces a data scarcity issue [1, 10, 11, 12, 24, 45, 60, 73].”
BackgroundiGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
Li, C., Xia, F., Martín-Martín, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.03272 · Cited by 58 · see citing works
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34 · see citing works
Honerkamp, D., Welschehold, T., & Valada, A. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3284346 · Cited by 30 · see citing works
Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.13724 · Cited by 15 · see citing works
Self-citationPhone2Proc: Bringing Robust Robots into Our Chaotic World
Deitke, M., Hendrix, R., Farhadi, A., Ehsani, K., & Kembhavi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00932 · Cited by 11 · see citing works
Self-citationLi, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., Martín-Martín, R., Wang, C., Levine, G., Ai, W., Martinez, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.09227 · Cited by 6 · see citing works
“To facilitate our experiments, we adopt an assistive pick primitive that creates a rigid connection between the object and the gripper if grasping is requested when all fingers are in contact with the object, a stricter form of StickyMitten used in prior works [26, 63, 64].”
MethodEvaluating Real-World Robot Manipulation Policies in Simulation
Li, X., Hsu, K., Gu, J., Pertsch, K., Mees, O., Walke, H. R., Fu, C., Lunawat, I., Sieh, I., Kirmani, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.05941 · Cited by 4 · see citing works
“That is, if one policy performs better than another in real-world 1] , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality of simulated evaluation pipelines.”
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., & Fox, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10721 · Cited by 7 · see citing works
- Authors
- Ehsani, K., Han, W., Herrasti, A., VanderBilt, E., Weihs, L., Kolve, E., Kembhavi, A., & Mottaghi, R.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 4495-4504
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3153322041
- Cited by
- 81
- References
- 3
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 20 · 2023: 24 · 2022: 12 · 2021: 4
- FWCI
- 22.07
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- object manipulation, embodied AI, AI2-THOR, mobile robots, grasping, occlusion, multi-object manipulation, long-horizon planning, generalization to unseen environments, physics simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
