CoChat
Sign Up
PreprintCited by 23Robot Manipulation and LearningMultimodal Machine Learning ApplicationsAI-based Problem Solving and Planning

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Embodied Task Planning with Large Language Models

Open access
Wu, Zhenyu, Wang, Ziwei, Xu, Xiuwei, Lu, Jiwen, Yan, Haibin·2023·arXiv
DOI 10.48550/arxiv.2307.01848arXiv 2307.01848
Open this paper in CoChat — free accountOpen PDF

Abstract

Equipping embodied agents with commonsense is important for robots to successfully complete complex human instructions in general environments. Recent large language models (LLM) can embed rich semantic knowledge for agents in plan generation of complex tasks, while they lack the information about the realistic world and usually yield infeasible action sequences. In this paper, we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models. Specifically, we first construct a multimodal dataset containing triplets of indoor scenes, instructions and action plans, where we provide the designed prompts and the list of existing objects in the scene for GPT-3.5 to generate a large number of instructions and corresponding planned actions. The generated data is leveraged for grounded plan tuning of pre-trained LLMs. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin, which indicates the practicality of embodied task planning in general and complex environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.01848

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In BSC-Nav demonstrations, natural language instructions are interpreted by GPT-4 to generate waypoint-action sequences.

    “In our demonstrations, natural language instructions are interpreted by GPT-464 to generate waypoint-action sequences.”

    From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [64]

    The Conclusion (section 5; page unavailable) says TaPA generates action steps from human instructions for robot navigation and manipulation, supporting the general instruction-to-plan idea. It says the dataset was generated with GPT-3.5, while the Experiment section (section 4; page unavailable) identifies a finetuned LLaMA-7B model as the task planner. These passages do not support the specific attribution to GPT-4 or waypoint-action sequences in BSC-Nav demonstrations. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [1][1] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658 , 2023. - · arXiv
  2. [2][2] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890 , 2023. - · arXiv
  3. [3][3] Z. Zhao, S. Wang, J. Gu, Y. Zhu, L. Mei, Z. Zhuang, Z. Cui, Q. Wang, and D. Shen. Chatcad+: Towards a universal and reliable interactive cad using llms. arXiv preprint arXiv:2305.15964 , 2023. - · arXiv
  4. [4][4] Y. Sun, C. Zhu, S. Zheng, K. Zhang, Z. Shui, X. Yu, Y. Zhao, H. Li, Y. Zhang, R. Zhao, et al. Pathasst: Redefining pathology through generative foundation ai assistant for pathology. arXiv preprint arXiv:2305.15072 , 2023. - · arXiv
  5. [6][6] F. Stella, C. Della Santina, and J. Hughes. How can llms transform the robotic design process? Nature Machine Intelligence , pages 1-4, 2023. -
  6. [8][8] R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao. Llamaadapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023. - · arXiv
  7. [9][9] B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 , 2023. - · arXiv
  8. [10][10] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023. - · arXiv
  9. [11][11] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022. -
  10. [12][12] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022. - · arXiv
  11. [13][13] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning , pages 991-1002. PMLR, 2022. -
  12. [14][14] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning , pages 287-318. PMLR, 2023. -
  13. [15][15] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. Llm-planner: Fewshot grounded planning for embodied agents with large language models. arXiv preprint arXiv:2212.04088 , 2022. - · arXiv
  14. [16][16] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10740-10749, 2020. -
  15. [17][17] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023. - · arXiv
  16. [18][18] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  17. [19][19] J. D. M.-W. C. Kenton and L. K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT , volume 1, page 2, 2019. -
  18. [20][20] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. - · arXiv
  19. [21][21] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023. - · arXiv
  20. [22][22] H. Bangalath, M. Maaz, M. U. Khattak, S. H. Khan, and F. Shahbaz Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. Advances in Neural Information Processing Systems , 35:33781-33794, 2022. -
  21. [23][23] X. Zhou, R. Girdhar, A. Joulin, P. Kr¨ ahenb¨ uhl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 350-368. Springer, 2022. -
  22. [25][25] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023. - · arXiv
  23. [26][26] F. Sammani, T. Mukherjee, and N. Deligiannis. Nlx-gpt: A model for natural language explanations in vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8322-8332, 2022. -
  24. [27][27] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023. - · arXiv
  25. [29][29] M. Zong and B. Krishnamachari. Solving math word problems concerning systems of equations with gpt-3. In Proceedings of the Thirteenth AAAI Symposium on Educational Advances in Artificial Intelligence , 2022. -
  26. [30][30] Y. Zang, W. Li, J. Han, K. Zhou, and C. C. Loy. Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305.18279 , 2023. - · arXiv
  27. [31][31] G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112.12143 , 2021. - · arXiv
  28. [32][32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. -
  29. [33][33] Z. Wu, Z. Wang, Z. Wei, Y. Wei, and H. Yan. Smart explorer: Recognizing objects in dense clutter via interactive exploration. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6600-6607. IEEE, 2022. -
  30. [34][34] Z. Liu, Z. Wang, S. Huang, J. Zhou, and J. Lu. Ge-grasp: Efficient target-oriented grasping in dense clutter. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1388-1395. IEEE, 2022. -
  31. [35][35] X. Xu, Z. Sun, Z. Wang, H. Liu, J. Zhou, and J. Lu. Dspdet3d: Dynamic spatial pruning for 3d small object detection. arXiv preprint arXiv:2305.03716 , 2023. - · arXiv
  32. [36][36] X. Xu, Y. Wang, Y. Zheng, Y. Rao, J. Zhou, and J. Lu. Back to reality: Weakly-supervised 3d object detection with shape-guided label enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8438-8447, 2022. -
  33. [37][37] V. Blukis, C. Paxton, D. Fox, A. Garg, and Y. Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In Conference on Robot Learning , pages 706-717. PMLR, 2022. -
  34. [38][38] R. Zellers, A. Holtzman, M. Peters, R. Mottaghi, A. Kembhavi, A. Farhadi, and Y. Choi. Piglet: Language grounding through neuro-symbolic interaction in a 3d world. arXiv preprint arXiv:2106.00188 , 2021. - · arXiv
  35. [39][39] A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud. Grounding language to autonomously-acquired skills via goal generation. arXiv preprint arXiv:2006.07185 , 2020. - · arXiv
  36. [40][40] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022. -
  37. [41][41] S. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Aky¨ urek, A. Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems , 35:31199-31212, 2022. -
  38. [42][42] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8494-8502, 2018. -
  39. [43][43] B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726 , 2023. - · arXiv
  40. [44][44] C. Lyu, M. Wu, L. Wang, X. Huang, B. Liu, Z. Du, S. Shi, and Z. Tu. Macaw-llm: Multimodal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093 , 2023. - · arXiv
  41. [45][45] Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplugowl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 , 2023. - · arXiv
  42. [46][46] D. Gao, L. Ji, L. Zhou, K. Q. Lin, J. Chen, Z. Fan, and M. Z. Shou. Assistgpt: A general multimodal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640 , 2023. - · arXiv
  43. [47][47] Z. Zhao, L. Guo, T. Yue, S. Chen, S. Shao, X. Zhu, Z. Yuan, and J. Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint arXiv:2305.16103 , 2023. - · arXiv
  44. [48][48] F. Chen, M. Han, H. Zhao, Q. Zhang, J. Shi, S. Xu, and B. Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160 , 2023. - · arXiv
  45. [49][49] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 , 2017. - · arXiv
  46. [50][50] X.-Y. Jiang, N.-Y. Pa, W.-C. Wang, T.-T. Yang, and W.-T. Pan. Site selection and layout of earthquake rescue center based on k-means clustering and fruit fly optimization algorithm. In 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) , pages 1381-1389. IEEE, 2020. -
  47. [51][51] X. Liu. The site selection of distribution center based on linear programming transportation method. In Proceedings of the 10th World Congress on Intelligent Control and Automation , pages 3538-3542. IEEE, 2012. Table 5: Our prompt for multimodal dataset generation (upper) and training/inference of TaPA (bottom). X a is empty unless the prompt serves as a ground-truth. <!-- image -->

Cited by

Sorted by citations

Metadata

Authors
Wu, Z., Wang, Z., Xu, X., Lu, J., & Yan, H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.01848
arXiv
2307.01848
OpenAlex
W4383604274
Cited by
23
References
2
Citations by year
2025: 9 · 2024: 13 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
large language models, multimodal dataset, object detection, open-vocabulary object detection, multimodal grounding, LLM fine-tuning, GPT-3.5, LLaVA, indoor scenes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review