CoChat
Sign Up
Cited by 96Robot Manipulation and LearningAI-based Problem Solving and PlanningSpeech and dialogue systems

ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application

Open access
Wake, Naoki, Kanehira, Atsushi, Sasabuchi, Kazuhiro, Takamatsu, Jun, Ikeuchi, Katsushi·2023·IEEE Access
DOI 10.1109/access.2023.3310935
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper introduces a novel method for translating natural-language instructions into executable robot actions using OpenAI’s ChatGPT in a few-shot setting. We propose customizable input prompts for ChatGPT that can easily integrate with robot execution systems or visual recognition programs, adapt to various environments, and create multi-step task plans while mitigating the impact of token limit imposed on ChatGPT. In our approach, ChatGPT receives both instructions and textual environmental data, and outputs a task plan and an updated environment. These environmental data are reused in subsequent task planning, thus eliminating the extensive record-keeping of prior task plans within the prompts of ChatGPT. Experimental results demonstrated the effectiveness of these prompts across various domestic environments, such as manipulations in front of a shelf, a fridge, and a drawer. The conversational capability of ChatGPT allows users to adjust the output via natural-language feedback. Additionally, a quantitative evaluation using VirtualHome showed that our results are comparable to previous studies. Specifically, 36% of task planning met both executability and correctness, and the rate approached 100% after several rounds of feedback. Our experiments revealed that ChatGPT can reasonably plan tasks and estimate post-operation environments without actual experience in object manipulation. Despite the allure of ChatGPT-based task planning in robotics, a standardized methodology remains elusive, making our work a substantial contribution. These prompts can serve as customizable templates, offering practical resources for the robotics research community. Our prompts and source code are open source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-Prompts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2023.3310935

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Maintaining a list of previously expanded nodes as additional Memory input facilitates a Markovian decision-making process and allows SayPlan to scale to extensive search sequences without maintaining full interaction history.

    “we maintain a list of previously expanded nodes, passed as an additional Memory input to the LLM, facilitating a Markovian decision-making process and allowing SayPlan to scale to extensive search sequences without the overhead of maintaining the full interaction history [5].”

    From SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning (2023) · cites this paper as [5]

    The ‘Specific Instruction from the User’ section (page unavailable) says that reusing the updated environment reduces the need for extensive historical records. The ‘Token Limit’ section (page unavailable) likewise says this reduces the burden of maintaining lengthy conversation histories, but warns that scalability remains limited. Neither section describes a list of previously expanded nodes, a Memory input, Markovian decision-making, or SayPlan’s ability to scale to extensive search sequences. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., ''Language models are few-shot learners,'' 2020 Advances in neural information processing systems (NeurIPS), vol. 33, pp. 1877-1901, 2020.
  2. P. Pramanick, H. B. Barua, and C. Sarkar, ''Decomplex: Task plan- ning from complex natural instructions by a collocating robot,'' in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6894-6901, IEEE, 2020.
  3. S. G. Venkatesh, R. Upadrashta, and B. Amrutur, ''Translating natural language instructions to computer programs for robot manipulation,'' in 2021 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp. 1919-1926, IEEE, 2021.
  4. I. Yanaokura, N. Wake, K. Sasabuchi, R. Arakawa, K. Okada, J. Takamatsu, M. Inaba, and K. Ikeuchi, ''A multimodal learning-from-observation to- wards all-at-once robot teaching using task cohesion,'' in 2022 IEEE/SICE International Symposium on System Integration (SII), pp. 367-374, IEEE, 2022.
  5. Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, ''Vima: General robot manipulation with multimodal prompts,'' arXiv preprint arXiv:2210.03094, 2022. · arXiv
  6. M. Shridhar, L. Manuelli, and D. Fox, ''Perceiver-actor: A multi-task transformer for robotic manipulation,'' in Conference on Robot Learning, pp. 785-799, PMLR, 2023.
  7. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al., ''Inner monologue: Embod- ied reasoning through planning with language models,'' arXiv preprint arXiv:2207.05608, 2022. · arXiv
  8. Y. Ding, X. Zhang, C. Paxton, and S. Zhang, ''Task and motion planning with large language models for object rearrangement,'' arXiv preprint arXiv:2303.06247, 2023. · arXiv
  9. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, ''Progprompt: Generating situated robot task plans using large language models,'' in 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 11523-11530, IEEE, 2023.
  10. K. Namasivayam, H. Singh, V. Bindal, A. Tuli, V. Agrawal, R. Jain, P. Singla, and R. Paul, ''Learning neuro-symbolic programs for language guided robot manipulation,'' in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7973-7980, IEEE, 2023.
  11. Z. Zhao, W. S. Lee, and D. Hsu, ''Differentiable parsing and visual ground- ing of natural language instructions for object placement,'' in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11546- 11553, IEEE, 2023.
  12. Y. Ding, X. Zhang, S. Amiri, N. Cao, H. Yang, C. Esselink, and S. Zhang, ''Robot task planning and situation handling in open worlds,'' arXiv preprint arXiv:2210.01287, 2022. · arXiv
  13. S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex, ''Planning with large language models via corrective re-prompting,'' arXiv preprint arXiv:2211.09935, 2022. · arXiv
  14. A. K. Kovalev and A. I. Panov, ''Application of pretrained large lan- guage models in embodied artificial intelligence,'' Doklady Mathematics, vol. 106, pp. S85 -S90, 2022.
  15. M. A. Khan, M. Kenney, J. Painter, D. Kamale, R. Batista-Navarro, and A. Ghalamzan-E, ''Natural language robot programming: Nlp integrated with autonomous robotic grasping,'' arXiv preprint arXiv:2304.02993, 2023. · arXiv
  16. F. Kaynar, S. Rajagopalan, S. Zhou, and E. Steinbach, ''Remote task- oriented grasp area teaching by non-experts through interactive segmen- tation and few-shot learning,'' arXiv preprint arXiv:2303.10195, 2023. · arXiv
  17. M. Skreta, N. Yoshikawa, S. Arellano-Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg, ''Errors are useful prompts: Instruction guided task programming with verifier-assisted itera- tive prompting,'' arXiv preprint arXiv:2303.14100, 2023. · arXiv
  18. J. Pan, G. Chou, and D. Berenson, ''Data-efficient learning of natural language to linear temporal logic translators for robot task specification,'' arXiv preprint arXiv:2303.08006, 2023. · arXiv
  19. K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, ''Text2motion: From natural language instructions to feasible plans,'' arXiv preprint arXiv:2303.12153, 2023. · arXiv
  20. C. Zhao, S. Yuan, C. Jiang, J. Cai, H. Yu, M. Y. Wang, and Q. Chen, ''Erra: An embodied representation and reasoning architecture for long-horizon language-conditioned manipulation tasks,'' IEEE Robotics and Automation Letters, 2023.
  21. H. Liu, L. Lee, K. Lee, and P. Abbeel, ''Instruction-following agents with multimodal transformer,'' arXiv preprint arXiv:2210.13431, 2022. · arXiv
  22. O. Mees, J. Borja-Diaz, and W. Burgard, ''Grounding language with visual affordances over unstructured data,'' in 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 11576-11582, IEEE, 2023.
  23. S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, ''Chatgpt for robotics: Design principles and model abilities,'' Microsoft Auton. Syst. Robot. Res, vol. 2, p. 20, 2023.
  24. X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba, ''Vir- tualhome: Simulating household activities via programs,'' in 2018 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8494-8502, 2018.
  25. N. Wake, R. Arakawa, I. Yanokura, T. Kiyokawa, K. Sasabuchi, J. Taka- matsu, and K. Ikeuchi, ''A learning-from-observation framework: One- shot robot teaching for grasp-manipulation-release household operations,'' in 2021 IEEE/SICE International Symposium on System Integration (SII), IEEE, 2021.
  26. K. Ikeuchi, N. Wake, R. Arakawa, K. Sasabuchi, and J. Takamatsu, ''Semantic constraints to represent common sense required in household actions for multi-modal learning-from-observation robot,'' arXiv preprint arXiv:2103.02201, 2021. · arXiv
  27. H. T. Kuhn and W. L. Inequalities, ''Related systems,'' Annals of Mathe- matic Studies, Princeton Univ. Press. EEUU, 1956.
  28. M. L. Minsky, ''Minsky's frame system theory,'' in Proceedings of the 1975 Workshop on Theoretical Issues in Natural Language Processing, TINLAP '75, (USA), p. 104-116, Association for Computational Linguistics, 1975.
  29. M. Gramopadhye and D. Szafir, ''Generating executable action plans with environmentally-aware language models,'' arXiv preprint arXiv:2210.04964, 2022. · arXiv
  30. N. Wake, K. Sasabuchi, and K. Ikeuchi, ''Grasp-type recognition leverag- ing object affordance,'' HOBI-RO-MAN Workshop, 2020.
  31. N. Wake, D. Saito, K. Sasabuchi, H. Koike, and K. Ikeuchi, ''Text-driven object affordance for guiding grasp-type recognition in multimodal robot teaching,'' Machine Vision and Applications, vol. 34, no. 4, p. 58, 2023.
  32. D. Saito, N. Wake, K. Sasabuchi, H. Koike, and K. Ikeuchi, ''Contact web status presentation for freehand grasping in mr-based robot-teaching,'' in Companion of the 2021 ACM/IEEE International Conference on Human- Robot Interaction, pp. 167-171, 2021.
  33. N. Wake, I. Yanokura, K. Sasabuchi, and K. Ikeuchi, ''Verbal focus-of- attention system for learning-from-demonstration,'' in 2021 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 10377-10384, IEEE, 2021.
  34. K. Sasabuchi, N. Wake, and K. Ikeuchi, ''Task-oriented motion mapping on robots of various configuration using body role division,'' IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 413-420, 2020.
  35. K. Ikeuchi, J. Takamatsu, K. Sasabuchi, N. Wake, and A. Kanehira, ''Applying learning-from-observation to household service robots: three common-sense formulations,'' arXiv preprint, 2023.
  36. N. Wake, M. Fukumoto, H. Takahashi, and K. Ikeuchi, ''Enhancing listen- ing capability of humanoid robot by reduction of stationary ego-noise,'' IEEJ Transactions on Electrical and Electronic Engineering, vol. 14, no. 12, pp. 1815-1822, 2019.
  37. J. Jaroslavceva, N. Wake, K. Sasabuchi, and K. Ikeuchi, ''Robot ego-noise suppression with labanotation-template subtraction,'' IEEJ Transactions on Electrical and Electronic Engineering, vol. 17, no. 3, pp. 407-415, 2022.
  38. N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, ''In- teractive task encoding system for learning-from-observation,'' in 2023 IEEE/ASME International Conference on Advanced Intelligent Mechatron- ics (AIM), pp. 1061-1066, 2023.
  39. J. Takamatsu, K. Sasabuchi, N. Wake, A. Kanehira, and K. Ikeuchi, ''Learning-from-observation system considering hardware-level reusabil- ity,'' arXiv preprint arXiv:2212.09242, 2022. · arXiv
  40. D. Saito, K. Sasabuchi, N. Wake, J. Takamatsu, H. Koike, and K. Ikeuchi, ''Task-grasping from a demonstrated human strategy,'' in 2022 IEEE- RAS 21st International Conference on Humanoid Robots (Humanoids), pp. 880-887, IEEE, 2022.
  41. K. Sasabuchi, D. Saito, A. Kanehira, N. Wake, J. Takamatsu, and K. Ikeuchi, ''Task-sequencing simulator: Integrated machine learn- ing to execution simulation for robot manipulation,'' arXiv preprint arXiv:2301.01382, 2023. · arXiv
  42. S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, ''Robots that use language,'' Annual Review of Control, Robotics, and Autonomous Systems, vol. 3, pp. 25-55, 2020.
  43. Y. Ye, H. You, and J. Du, ''Improved trust in human-robot collaboration with chatgpt,'' IEEE Access, vol. 11, pp. 55748-55754, 2023.
  44. Z. G. Cai, D. A. Haslett, X. Duan, S. Wang, and M. J. Pickering, ''Does chatgpt resemble humans in language use?,'' arXiv preprint arXiv:2303.08014, 2023. · arXiv
  45. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Ruther- ford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., ''Training compute-optimal large language models,'' arXiv preprint arXiv:2203.15556, 2022. · arXiv
  46. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., ''Llama 2: Open foundation and fine-tuned chat models,'' arXiv preprint arXiv:2307.09288, 2023. · arXiv

Cited by

Sorted by citations

Authors
Wake, N., Kanehira, A., Sasabuchi, K., Takamatsu, J., & Ikeuchi, K.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023
Volume
11
Pages
95060-95078
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2023.3310935
OpenAlex
W4386320380
Cited by
96
References
8
Citations by year
2026: 14 · 2025: 34 · 2024: 33 · 2023: 15
FWCI
10.30
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
ChatGPT, few-shot learning, task planning, multi-stage planning, prompt engineering, domestic environment, object manipulation, human-robot interaction, natural language feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review