The DOI matches its DataCite record.
Interactive Task Planning with Language Models
Open accessAbstract
An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional methods require predefined module design, making it hard to generalize to different goals. Recent large language model based approaches can allow for more open-ended planning but often require heavy prompt engineering or domain specific pretrained models. To tackle this, we propose a simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill execution through function calling, leveraging pretrained vision models to ground the scene in language. We verify the robustness of our system on the real world task of making milk tea drinks. Our system is able to generate novel high-level instructions for unseen objectives and successfully accomplishes user tasks. Furthermore, when the user sends a new request, our system is able to replan accordingly with precision based on the new request, task guidelines and previously executed steps. Our approach is easy to adapt to different tasks by simply substituting the task guidelines, without the need for additional complex prompt engineering. Please check more details on our https://wuphilipp.github.io/itp_site and https://youtu.be/TrKLuyv26_g.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can be used to combine multiple modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing, goal interpretation with subgoal decomposition, action sequencing with subgoal decomposition, and action sequencing with transition modeling.
“LLMs can also be used to combine multiple of the above modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing [13–32], goal interpretation with subgoal decomposition [2, 27, 33], action sequencing with subgoal decomposition [27, 34, 18, 35], action sequencing with transition modeling [8, 28, 32, 36, 37, 13, 38].”
From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 21
- Fahiem Bacchus and Qiang Yang. The downward refinement property. In Proceedings of the 12th International Joint Conference on Artificial Intelligence -Volume 1, IJCAI'91, pp. 286-292, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1558601600.
- Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence, 129(1):5-33, 2001.
- Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
- Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
- Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. In Proceedings of the 2nd International Joint Conference on Artificial Intelligence, IJCAI'71, pp. 608-620, San Francisco, CA, USA, 1971.
- Caelan Reed Garrett, Tomá s Lozano-Pérez, and Leslie Pack Kaelbling. FFRob: Leveraging symbolic planning for efficient task and motion planning. The International Journal of Robotics Research, 37(1):104-136, nov 2017. doi: 10.1177/0278364917739114. URL https://doi.org/10.1177%2F0278364917739114. · DOI
- Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: Theory and Practice. The Morgan Kaufmann Series in Artificial Intelligence. Morgan Kaufmann, Amsterdam, 2004. ISBN 978-1-55860-856-6. URL http://www.sciencedirect.com/science/book/9781558608566.
- Leslie Pack Kaelbling and Tomás Lozano-Pérez. Integrated task and motion planning in belief space. Int. J. Rob. Res., 32(9-10):1194-1227, aug 2013. ISSN 0278-3649. doi: 10.1177/0278364913484072. URL https://doi.org/10.1177/0278364913484072. · DOI
- Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pp. 1470-1477, 2011. doi: 10.1109/ICRA. 2011.5980391. · DOI
- Vladimir Lifschitz. What is answer set programming? In Proceedings of the 23rd National Conference on Artificial Intelligence -Volume 3, AAAI'08, pp. 1594-1597. AAAI Press, 2008. ISBN 9781577353683.
- Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13192-13200, June 2023. ISSN 2159-5399. doi: 10.1609/aaai.v37i11.26549. URL http://dx.doi.org/10.1609/aaai.v37i11.26549. · DOI
- Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023a. · arXiv
- Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724, 2023b. · arXiv
- Masoumeh Mansouri, Federico Pecora, and Peter Schüller. Combining task and motion planning: Challenges and guidelines. Frontiers in Robotics and AI, 8, 2021. ISSN 2296-9144. doi: 10.3389/frobt.2021.637888. URL https://www.frontiersin.org/articles/10.3389/frobt.2021.637888. OpenAI. Gpt-4 technical report, 2023. · DOI
- Erion Plaku and Gregory D. Hager. Sampling-based motion and symbolic action planning with geometric and differential constraints. In 2010 IEEE International Conference on Robotics and Automation, pp. 5002-5008, 2010. doi: 10.1109/ROBOT.2010.5509563. · DOI
- Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8494-8502, 2018.
- Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint arXiv:2307.06135, 2023. · arXiv
- Thierry Siméon, Jean-Paul Laumond, Juan Cortés, and Anis Sahbani. Manipulation planning with probabilistic roadmaps. The International Journal of Robotics Research, 23(7-8):729-746, 2004. doi: 10.1177/0278364904045471. URL https://doi.org/10.1177/0278364904045471. · DOI
- Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11530. IEEE, 2023.
- Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint arXiv:2303.14100, 2023. · arXiv
- Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291, 2023.
Cited by
Sorted by citations
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3188 · Cited by 21
Li, Z., Zhang, H., Peng, C., & Peiris, R. (2025) · 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr59515.2025.00025 · Cited by 24
Drift No More? Context Equilibria in Multi-Turn LLM Interactions
Dongre, V., Rossi, R. A., Lai, V. D., Yoon, D. S., Hakkani-Tür, D., & Bui, T. (2025) · arXiv · doi:10.48550/arxiv.2510.07777 · Cited by 3
Hippocampus: An Efficient and Scalable Memory Module for Agentic AI
Li, Y., Cao, L., Ahmed, F., Sharma, P., & Li, B. (2026) · arXiv · doi:10.48550/arxiv.2602.13594
“Without reliable recall, even sophisticated planners—such as those based on ReAct (Yao et al., 2023) or GoalAct (Chen et al., 2025)—can lose track of prior actions, repeat failed strategies, or misinterpret context (Li et al., 2023a; Xu et al., 2025).”
Self-citation
Metadata
- Authors
- Li, B., Wu, P., Abbeel, P., & Malik, J.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.10645
- OpenAlex
- W4387725621
- Cited by
- 41
- References
- 0
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotics and Automated Systems
- Keywords
- large language models, robot task planning, function calling, vision-language grounding, prompt engineering, long-horizon planning, replanning, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
