RobotGPT: Robot Manipulation Learning From ChatGPT
Open accessAbstract
We present RobotGPT, an innovative decision framework for robotic manipulation that prioritizes stability and safety. The execution code generated by ChatGPT cannot guarantee the stability and safety of the system. ChatGPT may provide different answers for the same task, leading to unpredictability. This instability prevents the direct integration of ChatGPT into the robot manipulation loop. Although setting the temperature to 0 can generate more consistent outputs, it may cause ChatGPT to lose diversity and creativity. Our objective is to leverage ChatGPT's problem-solving capabilities in robot manipulation and train a reliable agent. The framework includes an effective prompt structure and a robust learning model. Additionally, we introduce a metric for measuring task difficulty to evaluate ChatGPT's performance in robot manipulation. Furthermore, we evaluate RobotGPT in both simulation and real-world environments. Compared to directly using ChatGPT to generate code, our framework significantly improves task success rates, with an average increase from 38.5% to 91.5%. Therefore, training a RobotGPT by utilizing ChatGPT as an expert is a more stable approach compared to directly using ChatGPT as a task planner.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs and VLMs often produce physically impossible or inefficient actions due to affordance errors and causal real-world reasoning limitations that cause illogical or looping behaviors.
“LLMs and VLMs often produce physically impossible or inefficient actions due to affordance errors (incorrect reasoning about possible object actions) (Ahn et al., 2022; Li et al., 2025; Hu et al., 2024; Huang et al., 2022a; Jin et al., 2024) and causal real-world reasoning limitations that cause illogical or looping behaviors (Jin et al., 2024; Hu et al., 2024).”
From Large Language Model Reasoning Failures (2026) · cites this paper as Jin et al., 2024
In “ChatGPT prompts for robot manipulation” (page not provided), the paper says ChatGPT-generated actions may disregard physical constraints and that ChatGPT lacks causal and temporal reasoning. “Experiment setup” (page not provided) also describes unreasonable action sequencing. The paper does not establish the claim about VLMs, affordance errors, inefficiency, or looping behavior. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · see citing works
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624 · see citing works
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593986 · Cited by 606 · see citing works
ChatGPT for Robotics: Design Principles and Model Abilities
Vemprala, S. H., Bonatti, R., Bucker, A., & Kapoor, A. (2024) · IEEE Access · doi:10.1109/access.2024.3387941 · Cited by 570 · see citing works
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, "Text2motion: From natural language instructions to fea- sible plans," in ICRA2023 Workshop on Pretraining for Robotics (PT4R), 2023.
- M. Marge, C. Espy-Wilson, N. G. Ward, A. Al- wan, Y. Artzi, M. Bansal, G. Blankenship, J. Chai, H. Daumé III, D. Dey, et al., "Spoken language interac- tion with robots: Recommendations for future research," Computer Speech & Language, vol. 71, p. 101255, 2022.
- S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, "Robots that use language," Annual Review of Control, Robotics, and Autonomous Systems, vol. 3, pp. 25-55, 2020.
- T. Kollar, S. Tellex, D. Roy, and N. Roy, "Toward understanding natural language directions," in 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 259-266, IEEE, 2010.
- S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al., "Learning language-conditioned robot behavior from of- fline data and crowd-sourced annotation," in Conference on Robot Learning, pp. 1303-1315, PMLR, 2022.
- A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig, "Language models of code are few-shot commonsense learners," arXiv preprint arXiv:2210.07128, 2022. · arXiv
- J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, "Tidybot: Personalized robot assistance with large lan- guage models," arXiv preprint arXiv:2305.05658, 2023. · arXiv
- O. Kroemer, S. Niekum, and G. Konidaris, "A review of robot learning for manipulation: Challenges, representa- tions, and algorithms," The Journal of Machine Learning Research, vol. 22, no. 1, pp. 1395-1476, 2021.
- S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, et al., "Scaling data-driven robotics with reward sketching and batch reinforcement learning," arXiv preprint arXiv:1909.12200, 2019. · arXiv
- D. Wang, R. Walters, X. Zhu, and R. Platt, "Equivariant q learning in spatial action spaces," in Conference on Robot Learning, pp. 1713-1723, PMLR, 2022.
- A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al., "Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching," The International Journal of Robotics Re- search, vol. 41, no. 7, pp. 690-705, 2022.
- Y. Lee, E. S. Hu, and J. J. Lim, "Ikea furniture assem- bly environment for long-horizon complex manipulation tasks," in 2021 ieee international conference on robotics and automation (icra), pp. 6343-6349, IEEE, 2021.
- Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, and Y. Zhu, "robosuite: A modular simulation framework and benchmark for robot learning," arXiv preprint arXiv:2009.12293, 2020. · arXiv
- B. Delhaisse, L. Rozo, and D. G. Caldwell, "Pyrobolearn: A python framework for robot learning practitioners," in Conference on Robot Learning, pp. 1348-1358, PMLR, 2020.
- S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, "Rlbench: The robot learning benchmark & learning environment," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3019-3026, 2020.
- D. Wang, C. Kohler, X. Zhu, M. Jia, and R. Platt, "Bul- letarm: An open-source robotic manipulation benchmark and learning framework," in Robotics Research, pp. 335- 350, Springer, 2023.
- D. Wang, C. Kohler, and R. Platt, "Policy learning in se (3) action spaces," arXiv preprint arXiv:2010.02798, 2020. · arXiv
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Ve- ness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., "Human-level control through deep reinforcement learning," nature, vol. 518, no. 7540, pp. 529-533, 2015.
- A. S. Lakshminarayanan, S. Ozair, and Y. Bengio, "Re- inforcement learning with few expert demonstrations," in NIPS workshop on deep learning for action and interaction, vol. 2016, 2016.
- T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Os- band, et al., "Deep q-learning from demonstrations," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.
- J. Long, E. Shelhamer, and T. Darrell, "Fully convolu- tional networks for semantic segmentation," in Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Cited by
Sorted by citations
Li, Z., Zhang, H., Peng, C., & Peiris, R. (2025) · 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr59515.2025.00025 · Cited by 18 · see citing works
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
- Authors
- Jin, Y., Li, D., A, Y., Shi, J., Hao, P., Sun, F., Zhang, J., & Fang, B.
- Affiliations
- Samsung Research China– Beijing (SRC-B), Beijing, China
- Tsinghua University (CN)
- Universität Hamburg (DE)
- Beijing University of Posts and Telecommunications (CN)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2024-03
- Volume
- 9
- Issue
- 3
- Pages
- 2543-2550
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4391128534
- Cited by
- 103
- References
- 5
- Citations by year
- 2026: 30 · 2025: 56 · 2024: 17
- FWCI
- 16.14
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- New Generation Artificial Intelligence
- Key Field Research and Development Plan of Guangdong Province (2021B0101410002)
- National Natural Science Foundation of China (62173197)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- robotic manipulation, ChatGPT, large language models, prompt engineering, task planning, code generation, robot safety, reinforcement learning from human feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
