Language Models as Zero-Shot Trajectory Generators
Open accessAbstract
Large Language Models (LLMs) have recently shown promise as high-level planners for robots when given access to a selection of low-level skills. However, it is often assumed that LLMs do not possess sufficient knowledge to be used for the low-level trajectories themselves. In this work, we address this assumption thoroughly, and investigate if an LLM (GPT-4) can directly predict a dense sequence of end-effector poses for manipulation tasks, when given access to only object detection and segmentation vision models. We designed a single, task-agnostic prompt, without any in-context examples, motion primitives, or external trajectory optimisers. Then we studied how well it can perform across 30 real-world language-based tasks, such as “open the bottle cap” and “wipe the plate with the sponge”, and we investigated which design choices in this prompt are the most important. Our conclusions raise the assumed limit of LLMs for robotics, and we reveal for the first time that LLMs do indeed possess an understanding of low-level robot control sufficient for a range of common tasks, and that they can additionally detect failures and then re-plan trajectories accordingly.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous work needed to compose multiple models to control a robot without robot action data.
“Whereas previous work needed to compose multiple models to this end (Ahn et al., 2022; Kwon et al., 2024; Liang et al., 2023; Vemprala et al., 2023)”
From Gemini Robotics: Bringing AI into the Physical World (2025) · cites this paper as Kwon et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 39 · Cited by 59
Research works printed but not linked: 25
- W. Huang et al., "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models," arXiv e-prints, p. arXiv:2307.05973, July 2023. · arXiv
- W. Yu et al., "Language to Rewards for Robotic Skill Synthesis," arXiv e-prints, p. arXiv:2306.08647, June 2023. · arXiv
- OpenAI, "GPT-4 Technical Report," arXiv e-prints, p. arXiv:2303.08774, Mar. 2023. · arXiv
- A. Brohan et al., "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control," arXiv e-prints, p. arXiv:2307.15818, July 2023. · arXiv
- J.-B. Alayrac et al., "Flamingo: a visual language model for few-shot learning," in Advances in Neural Information Processing Systems, S. Koyejo et al., Eds., vol. 35. Curran Associates, Inc., 2022, pp. 23 716-23 736. [On- line]. Available: https://proceedings.neurips.cc/paper files/paper/2022/ file/960a172bc7fbf0177ccccbb411a7d800-Paper-Conference.pdf
- N. Di Palo and E. Johns, "Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics," arXiv e-prints, p. arXiv:2403.19578, Mar. 2024. · arXiv
- S. Mirchandani et al., "Large Language Models as General Pattern Machines," arXiv e-prints, p. arXiv:2307.04721, July 2023. · arXiv
- W. Huang et al., "Inner Monologue: Embodied Reasoning through Planning with Language Models," arXiv e-prints, p. arXiv:2207.05608, July 2022. · arXiv
- N. Di Palo et al., "Towards A Unified Agent with Foundation Models," arXiv e-prints, p. arXiv:2307.09668, July 2023. · arXiv
- W. Shen et al., "Distilled Feature Fields Enable Few-Shot Language- Guided Manipulation," arXiv e-prints, p. arXiv:2308.07931, July 2023. · arXiv
- J. Kerr et al., "LERF: Language Embedded Radiance Fields," arXiv e-prints, p. arXiv:2303.09553, Mar. 2023. · arXiv
- A. Rashid et al., "Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping," arXiv e-prints, p. arXiv:2309.07970, Sept. 2023. · arXiv
- N. Di Palo and E. Johns, "DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models," arXiv e-prints, p. arXiv:2402.13181, Feb. 2024. · arXiv
- I. Kapelyukh et al., "DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics," arXiv e-prints, p. arXiv:2210.02438, Oct. 2022. · arXiv
- --, "Dream2Real: Zero-Shot 3D Object Rearrangement with Vision- Language Models," arXiv e-prints, p. arXiv:2312.04533, Dec. 2023. · arXiv
- S. Liu et al., "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection," arXiv e-prints, p. arXiv:2303.05499, Mar. 2023. · arXiv
- H. K. Cheng and A. G. Schwing, "Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model," in Computer Vision -ECCV 2022, S. Avidan et al., Eds. Cham: Springer Nature Switzerland, 2022, pp. 640-658.
- T. Xiao et al., "Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models," arXiv e-prints, p. arXiv:2211.11736, Nov. 2022. · arXiv
- A. Brohan et al., "RT-1: Robotics Transformer for Real-World Control at Scale," arXiv e-prints, p. arXiv:2212.06817, Dec. 2022. · arXiv
- T. Yu et al., "Scaling Robot Learning with Semantically Imagined Experience," arXiv e-prints, p. arXiv:2302.11550, Feb. 2023. · arXiv
- J. Wei et al., "Chain-of-thought prompting elicits reason- ing in large language models," in Advances in Neural Information Processing Systems, S. Koyejo et al., Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824-24 837. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
- T. Kojima et al., "Large language models are zero-shot reasoners," in Advances in Neural Information Processing Systems, S. Koyejo et al., Eds., vol. 35. Curran Associates, Inc., 2022, pp. 22 199-22 213. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf
- J. Hoffmann et al., "Training Compute-Optimal Large Language Mod- els," arXiv e-prints, p. arXiv:2203.15556, Mar. 2022. · arXiv
- H. Luo et al., "WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct," arXiv e-prints, p. arXiv:2308.09583, Aug. 2023. · arXiv
- H. Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv e-prints, p. arXiv:2307.09288, July 2023. · arXiv
- Authors
- Kwon, T., Palo, N. D., & Johns, E.
- Affiliations
- Imperial College London (GB)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2024-07
- Volume
- 9
- Issue
- 7
- Pages
- 6728-6735
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4399374221
- Cited by
- 59
- References
- 39
- Citations by year
- 2026: 18 · 2025: 31 · 2024: 10
- FWCI
- 9.25
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Royal Academy of Engineering
- Research Fellowship Scheme
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- large language models, robotic manipulation, GPT-4, prompt engineering, robot planning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
