CoChat
Sign Up
PreprintCited by 56Robot Manipulation and LearningAI-based Problem Solving and PlanningRobotics and Automated Systems

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Creative Robot Tool Use with Large Language Models

Open access
Xu, Mengdi, Huang, Peide, Yu, Wenhao, Liu, Shiqi, Zhang, Xilun, Niu, Yaru, Zhang, Tingnan, Xia, Fei, Tan, Jie, Zhao, Ding·2023·arXiv
DOI 10.48550/arxiv.2310.13065arXiv 2310.13065
Open this paper in CoChat — free accountOpen PDF

Abstract

Tool use is a hallmark of advanced intelligence, exemplified in both animal behavior and robotic capabilities. This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning. Leveraging Large Language Models (LLMs), we develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments. RoboTool incorporates four pivotal components: (i) an "Analyzer" that interprets natural language to discern key task-related concepts, (ii) a "Planner" that generates comprehensive strategies based on the language input and key concepts, (iii) a "Calculator" that computes parameters for each skill, and (iv) a "Coder" that translates these plans into executable Python code. Our results show that RoboTool can not only comprehend explicit or implicit physical constraints and environmental factors but also demonstrate creative tool use. Unlike traditional Task and Motion Planning (TAMP) methods that rely on explicit optimization, our LLM-based system offers a more flexible, efficient, and user-friendly solution for complex robotics tasks. Through extensive experiments, we validate that RoboTool is proficient in handling tasks that would otherwise be infeasible without the creative use of tools, thereby expanding the capabilities of robotic systems. Demos are available on our project page: https://creative-robotool.github.io/.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.13065

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can be used to combine multiple modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing, goal interpretation with subgoal decomposition, action sequencing with subgoal decomposition, and action sequencing with transition modeling.

    “LLMs can also be used to combine multiple of the above modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing [13–32], goal interpretation with subgoal decomposition [2, 27, 33], action sequencing with subgoal decomposition [27, 34, 18, 35], action sequencing with transition modeling [8, 28, 32, 36, 37, 13, 38].”

    From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [16]

    The citation to this paper appears in the action-sequencing-with-transition-modeling group. Section 3.2.2 (page unavailable) says RoboTool’s LLM Planner generates a sequence of plan steps, but the paper does not describe using the LLM to model state transitions; Section 3.1 merely includes transition dynamics in its general problem formulation. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. 2, 3, 5 · arXiv
  2. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 2, 3 Josep Call. Three ingredients for becoming a creative tool user, pp. 3-20. Cambridge University Press, 2013. doi: 10.1017/CBO9780511894800.002. · DOI
  3. Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Autotamp: Au- toregressive task and motion planning with llms as translators and checkers. arXiv preprint arXiv:2306.06531, 2023. 2, 3 · arXiv
  4. Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247, 2023. · arXiv
  5. Carl Henrik Ek, Dan Song, Kai Huebner, and Danica Kragic. Exploring affordances in robot grasp- ing through latent structure representation. Vision for Cognitive Tasks, ECCV, 2010.
  6. Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research, 39(2-3):202-216, 2020. 3, 17
  7. M. Fischler and R. Bolles. Random sample consensus: A paradigm for model fitting with applica- tions to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981. 15, 17
  8. Jonathan D. Gammell, Siddhartha S. Srinivasa, and Timothy D. Barfoot. Informed rrt*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2997-3004, 2014. doi: 10.1109/IROS.2014.6942976. · DOI
  9. Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pp. 440-448, 2020.
  10. Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021. 2, 3
  11. Michael D Gumert, Marius Kluck, and Suchinda Malaivijitnond. The physical characteristics and usage patterns of stone axe and pounding hammers used by long-tailed macaques in the andaman sea region of thailand. American Journal of Primatology: Official Journal of the American Society of Primatologists, 71(7):594-608, 2009.
  12. Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176, 2023a. · arXiv
  13. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 9118-9147. PMLR, 2022a. 2, 3, 5
  14. Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023b. 2, 3, 10 · arXiv
  15. Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303.00855, 2023c. · arXiv
  16. Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pp. 1470-1477. IEEE, 2011.
  17. Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6185-6191. IEEE, 2021.
  18. Wolfgang Kohler. The mentality of apes. Routledge, 2018.
  19. Martin Levihn and Mike Stilman. Using environment objects as tools: Unconventional door open- ing. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2502- 2508. IEEE, 2014.
  20. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023. 3, 6, 7
  21. Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153, 2023. 2, 3, 10 · arXiv
  22. Yun Lin and Yu Sun. Robot grasp planning based on demonstrated grasp strategies. The Interna- tional Journal of Robotics Research, 34(1):26-42, 2015.
  23. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477, 2023. · arXiv
  24. Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022. 7, 17, 18
  25. Katharina Muelling, Jens Kober, and Jan Peters. Learning table tennis with a mixture of motor primitives. In 2010 10th IEEE-RAS International Conference on Humanoid Robots, pp. 411-416. IEEE, 2010.
  26. Austin Myers, Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1374-1381. IEEE, 2015.
  27. Fusaomi Nagata, Keigo Watanabe, and Kiyotaka Izumi. Furniture polishing robot using a trajectory generator based on cutter location data. In Proceedings 2001 ICRA. IEEE International Confer- ence on Robotics and Automation (Cat. No. 01CH37164), volume 1, pp. 319-324. IEEE, 2001.
  28. Edwin Olson. Apriltag: A robust and flexible visual fiducial system. In 2011 IEEE international conference on robotics and automation, pp. 3400-3407. IEEE, 2011.
  29. Kai Pfeiffer, Adrien Escande, and Abderrahmane Kheddar. Nut fastening with a humanoid robot. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6142- 6148. IEEE, 2017.
  30. Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928, 2023a. · arXiv
  31. Allen Z Ren, Bharat Govil, Tsung-Yen Yang, Karthik R Narasimhan, and Anirudha Majumdar. Leveraging language for accelerated learning of tool manipulation. In Conference on Robot Learning, pp. 1531-1541. PMLR, 2023b.
  32. Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305.11014, 2023. · arXiv
  33. Jivko Sinapov and Alexadner Stoytchev. Detecting the functional similarities between tools using a hierarchical representation of outcomes. In 2008 7th IEEE International Conference on Develop- ment and Learning, pp. 91-96. IEEE, 2008.
  34. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11530. IEEE, 2023.
  35. Dan Song, Kai Huebner, Ville Kyrki, and Danica Kragic. Learning task constraints for robot grasp- ing using graphical models. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1579-1585. IEEE, 2010.
  36. Dan Song, Carl Henrik Ek, Kai Huebner, and Danica Kragic. Task-based robot grasp planning using probabilistic inference. IEEE transactions on robotics, 31(3):546-561, 2015.
  37. Yijia Weng, Kaichun Mo, Ruoxi Shi, Yanchao Yang, and Leonidas Guibas. Towards learning geo- metric eigen-lengths crucial for fitting tasks. ICML, 2023.
  38. Handy Wicaksono and Claude Sammut. Relational tool use learning by a robot in a real and simu- lated world. In Proceedings of ACRA, 2016.
  39. Annie Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical un- derstanding: Using novel objects as tools with visual foresight. arXiv preprint arXiv:1904.05538, 2019. · arXiv
  40. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural lan- guage to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023. · arXiv
  41. Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647, 2023. 2, 6 · arXiv
  42. Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Xu, M., Huang, P., Yu, W., Liu, S., Zhang, X., Niu, Y., Zhang, T., Xia, F., Tan, J., & Zhao, D.
Affiliations
  • Carnegie Mellon University (US)
  • Google DeepMind (United Kingdom) (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.13065
arXiv
2310.13065
OpenAlex
W4387891177
Cited by
56
References
1
Citations by year
2026: 20 · 2025: 20 · 2024: 16
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Engineering › Control and Systems Engineering › Robotics and Automated Systems
Keywords
large language models, robot control, natural language instructions, task and motion planning, long-term planning, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review