CoChat
Sign Up
PreprintCited by 146Robot Manipulation and LearningAI-based Problem Solving and PlanningNatural Language Processing Techniques

The DOI matches its DataCite record.

AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers

Open access
Chen, Yongchao, Arkin, Jacob, Dawson, Charles, Zhang, Yang, Roy, Nicholas, Fan, Chuchu·2023·arXiv
DOI 10.48550/arxiv.2306.06531arXiv 2306.06531
Open this paper in CoChat — free accountOpen PDF

Abstract

For effective human-robot interaction, robots need to understand, plan, and execute complex, long-horizon tasks described by natural language. Recent advances in large language models (LLMs) have shown promise for translating natural language into robot action sequences for complex tasks. However, existing approaches either translate the natural language directly into robot trajectories or factor the inference process by decomposing language into task sub-goals and relying on a motion planner to execute each sub-goal. When complex environmental and temporal constraints are involved, inference over planning tasks must be performed jointly with motion plans using traditional task-and-motion planning (TAMP) algorithms, making factorization into subgoals untenable. Rather than using LLMs to directly plan task sub-goals, we instead perform few-shot translation from natural language task descriptions to an intermediate task representation that can then be consumed by a TAMP algorithm to jointly solve the task and motion plan. To improve translation, we automatically detect and correct both syntactic and semantic errors via autoregressive re-prompting, resulting in significant improvements in task completion. We show that our approach outperforms several methods using LLMs as planners in complex task domains. See our project website https://yongchao98.github.io/MIT-REALM-AutoTAMP/ for prompts, videos, and code.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.06531

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can be used to combine multiple modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing, goal interpretation with subgoal decomposition, action sequencing with subgoal decomposition, and action sequencing with transition modeling.

    “LLMs can also be used to combine multiple of the above modules at once via chain-of-thought prompting or pipelined queries, such as goal interpretation with action sequencing [13–32], goal interpretation with subgoal decomposition [2, 27, 33], action sequencing with subgoal decomposition [27, 34, 18, 35], action sequencing with transition modeling [8, 28, 32, 36, 37, 13, 38].”

    From Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (2024) · cites this paper as [18]

    Section C describes a pipeline in which an LLM interprets a natural-language task by translating it into STL, after which a formal planner produces a trajectory. The “STL Trajectory Planner” section says that planner searches for a sub-task sequence under dynamical constraints. This supports the goal-interpretation/action-sequencing example, but not the attribution that the LLM combines action sequencing with transition modeling; the paper assigns sequence search to the separate planner. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 59

  1. C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kael- bling, and T. Lozano-Pérez, "Integrated task and motion planning," Annual review of control, robotics, and autonomous systems, vol. 4, pp. 265-293, 2021.
  2. M. Fox and D. Long, "Pddl2. 1: An extension to pddl for expressing temporal planning domains," Journal of artificial intelligence research, vol. 20, pp. 61-124, 2003.
  3. E. A. Emerson, "Temporal and modal logic," in Formal Models and Semantics. Elsevier, 1990, pp. 995-1072.
  4. K. He, M. Lahijanian, L. E. Kavraki, and M. Y. Vardi, "Towards manipulation planning with temporal logic specifications," in 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 346-352.
  5. R. E. Fikes and N. J. Nilsson, "Strips: A new approach to the application of theorem proving to problem solving," Artificial Intelligence, vol. 2, no. 3, pp. 189-208, 1971. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0004370271900105
  6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language models are few-shot learners," Advances in neural information pro- cessing systems, vol. 33, pp. 1877-1901, 2020.
  7. J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, "Tidybot: Personal- ized robot assistance with large language models," arXiv preprint arXiv:2305.05658, 2023. · arXiv
  8. W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, "Language models as zero-shot planners: Extracting actionable knowledge for embodied agents," in International Conference on Machine Learning. PMLR, 2022, pp. 9118-9147.
  9. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al., "Do as i can, not as i say: Grounding language in robotic affordances," arXiv preprint arXiv:2204.01691, 2022. · arXiv
  10. I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, "ProgPrompt: Generating situated robot task plans using large language models," in International Conference on Robotics and Automation (ICRA), 2023. [Online]. Available: https://arxiv.org/abs/2209.11302 · arXiv
  11. W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al., "Inner monologue: Embodied reasoning through planning with language models," arXiv preprint arXiv:2207.05608, 2022. · arXiv
  12. K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, "Text2motion: From natural language instructions to feasible plans," arXiv preprint arXiv:2303.12153, 2023. · arXiv
  13. Y. Ding, X. Zhang, C. Paxton, and S. Zhang, "Task and motion planning with large language models for object rearrangement," arXiv preprint arXiv:2303.06247, 2023. · arXiv
  14. N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, "Chatgpt empowered long-step robot control in various environments: A case application," arXiv preprint arXiv:2304.03893, 2023. · arXiv
  15. K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati, "Large language models still can't plan (a benchmark for llms on planning and reasoning about change)," arXiv preprint arXiv:2206.10498, 2022. · arXiv
  16. B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone, "Llm+ p: Empowering large language models with optimal planning proficiency," arXiv preprint arXiv:2304.11477, 2023. · arXiv
  17. Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh, "Translating natural language to planning goals with large-language models," arXiv preprint arXiv:2302.05128, 2023. · arXiv
  18. J. Pan, G. Chou, and D. Berenson, "Data-efficient learning of natural language to linear temporal logic translators for robot task specifica- tion," arXiv preprint arXiv:2303.08006, 2023. · arXiv
  19. C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling, "Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning," in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 30, 2020, pp. 440-448.
  20. M. Skreta, N. Yoshikawa, S. Arellano-Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg, "Errors are useful prompts: Instruction guided task programming with verifier- assisted iterative prompting," arXiv preprint arXiv:2303.14100, 2023. · arXiv
  21. Y. Chen, R. Gandhi, Y. Zhang, and C. Fan, "Nl2tl: Transforming natural languages to temporal logics using large language models," arXiv preprint arXiv:2305.07766, 2023. · arXiv
  22. O. Maler and D. Nickovic, "Monitoring temporal properties of con- tinuous signals," in Formal Techniques, Modelling and Analysis of Timed and Fault-Tolerant Systems: Joint International Conferences on Formal Modeling and Analysis of Timed Systmes, FORMATS 2004, and Formal Techniques in Real-Time and Fault-Tolerant Systems, FTRTFT 2004, Grenoble, France, September 22-24, 2004. Proceed- ings. Springer, 2004, pp. 152-166.
  23. D. Sun, J. Chen, S. Mitra, and C. Fan, "Multi-agent motion plan- ning from signal temporal logic specifications," IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3451-3458, 2022.
  24. C. Finucane, G. Jing, and H. Kress-Gazit, "Ltlmop: Experimenting with language, temporal logic and robot control," in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2010, pp. 1988-1993.
  25. S. Wilson, P. Glotfelter, L. Wang, S. Mayya, G. Notomista, M. Mote, and M. Egerstedt, "The robotarium: Globally impactful opportunities, challenges, and lessons learned in remote-access, distributed control of multirobot systems," IEEE Control Systems Magazine, vol. 40, no. 1, pp. 26-44, 2020.
  26. S. M. LaValle, Planning algorithms. Cambridge university press, 2006.
  27. J. Ferrer-Mestres, G. Frances, and H. Geffner, "Combined task and motion planning as classical ai planning," arXiv preprint arXiv:1706.06927, 2017. · arXiv
  28. C. R. Garrett, T. Lozano-Perez, and L. P. Kaelbling, "Ffrob: Lever- aging symbolic planning for efficient task and motion planning," The International Journal of Robotics Research, vol. 37, no. 1, pp. 104- 136, 2018.
  29. A. Akbari, J. Rosell, et al., "Task planning using physics-based heuris- tics on manipulation actions," in 2016 IEEE 21st International Con- ference on Emerging Technologies and Factory Automation (ETFA). IEEE, 2016, pp. 1-8.
  30. F. Lagriffoul and B. Andres, "Combining task and motion planning: A culprit detection problem," The International Journal of Robotics Research, vol. 35, no. 8, pp. 890-927, 2016.
  31. J. Wolfe, B. Marthi, and S. Russell, "Combined task and motion planning for mobile manipulation," in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 20, 2010, pp. 254-257.
  32. S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel, "Combined task and motion planning through an extensible planner- independent interface layer," in 2014 IEEE international conference on robotics and automation (ICRA). IEEE, 2014, pp. 639-646.
  33. M. Colledanchise, D. Almeida, and P. Ögren, "Towards blended reac- tive planning and acting using behavior trees," in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 8839-8845.
  34. L. P. Kaelbling and T. Lozano-Pérez, "Integrated task and motion planning in belief space," The International Journal of Robotics Research, vol. 32, no. 9-10, pp. 1194-1227, 2013.
  35. E. Fernandez-Gonzalez, B. Williams, and E. Karpas, "Scottyactivity: Mixed discrete-continuous planning with convex optimization," Jour- nal of Artificial Intelligence Research, vol. 62, pp. 579-664, 2018.
  36. K. He, M. Lahijanian, L. E. Kavraki, and M. Y. Vardi, "Towards manipulation planning with temporal logic specifications," in 2015 IEEE international conference on robotics and automation (ICRA). IEEE, 2015, pp. 346-352.
  37. M. Katayama, S. Tokuda, M. Yamakita, and H. Oyama, "Fast ltl- based flexible planning for dual-arm manipulation," in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 6605-6612.
  38. R. Takano, H. Oyama, and M. Yamakita, "Continuous optimization- based task and motion planning with signal temporal logic speci- fications for sequential manipulation," in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 8409-8415.
  39. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot reasoners," in ICML 2022 Workshop on Knowledge Retrieval and Language Models, 2022. [Online]. Available: https://openreview.net/forum?id=6p3AuaHAFiN
  40. T. Silver, V. Hariprasad, R. S. Shuttleworth, N. Kumar, T. Lozano- Pérez, and L. P. Kaelbling, "PDDL planning with pretrained large language models," in NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. [Online]. Available: https: //openreview.net/forum?id=1QMMUB4zfl
  41. L. S. Zettlemoyer and M. Collins, "Learning to map sentences to logical form: structured classification with probabilistic categorial grammars," in Proceedings of the Twenty-First Conference on Un- certainty in Artificial Intelligence, 2005, pp. 658-666.
  42. L. Zettlemoyer and M. Collins, "Online learning of relaxed ccg grammars for parsing to logical form," in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 678-687.
  43. Y. W. Wong and R. J. Mooney, "Learning for semantic parsing with statistical machine translation," in Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. Association for Computational Linguistics, 2006, pp. 439-446.
  44. J. Dzifcak, M. Scheutz, C. Baral, and P. Schermerhorn, "What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution," in 2009 IEEE International Conference on Robotics and Automation. IEEE, 2009, pp. 4163-4168.
  45. Y. Artzi and L. Zettlemoyer, "Weakly supervised learning of semantic parsers for mapping instructions to actions," Transactions of the Association for Computational Linguistics, vol. 1, pp. 49-62, 2013.
  46. T. M. Howard, S. Tellex, and N. Roy, "A natural language planner interface for mobile manipulators," in 2014 IEEE International Con- ference on Robotics and Automation (ICRA). IEEE, 2014, pp. 6652- 6659.
  47. A. Boteanu, J. Arkin, T. Howard, and H. Kress-Gazit, "A model for verifiable grounding and execution of complex language instructions," in Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 2016.
  48. N. Gopalan, D. Arumugam, L. L. Wong, and S. Tellex, "Sequence-to- sequence language grounding of non-markovian task specifications." in Robotics: Science and Systems, vol. 2018, 2018.
  49. R. Patel, E. Pavlick, and S. Tellex, "Grounding language to non- markovian tasks with no supervision of task specifications." in Robotics: Science and Systems, vol. 2020, 2020.
  50. H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas, "Translating structured english to robot controllers," Advanced Robotics, vol. 22, no. 12, pp. 1343-1359, 2008.
  51. J. He, E. Bartocci, D. Ničković, H. Isakovic, and R. Grosu, "Deepstl: from english requirements to signal temporal logic," in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 610-622.
  52. S. Mohammadinejad, J. Thomason, and J. V. Deshmukh, "Interactive learning from natural language and demonstrations using signal tem- poral logic," arXiv preprint arXiv:2207.00627, 2022. · arXiv
  53. C. N. Bonial, L. Donatelli, J. Ervin, and C. R. Voss, "Abstract meaning representation for human-robot dialogue," Proceedings of the Society for Computation in Linguistics, vol. 2, no. 1, pp. 236-246, 2019.
  54. S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, "Robots that use language," Annual Review of Control, Robotics, and Autonomous Systems, vol. 3, pp. 25-55, 2020.
  55. J. X. Liu, Z. Yang, B. Schornstein, S. Liang, I. Idrees, S. Tellex, and A. Shah, "Lang2LTL: Translating natural language commands to temporal specification with large language models," in Workshop on Language and Robotics at CoRL 2022, 2022. [Online]. Available: https://openreview.net/forum?id=VxfjGZzrdn
  56. L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati, "Leveraging pre-trained large language models to construct and uti- lize world models for model-based task planning," arXiv preprint arXiv:2305.14909, 2023. · arXiv
  57. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, "Code as policies: Language model programs for embodied control," arXiv preprint arXiv:2209.07753, 2022. · arXiv
  58. S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex, "Planning with large language models via corrective re-prompting," in NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. [Online]. Available: https://openreview.net/ forum?id=cMDMRBe1TKs
  59. T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. P. Kaelbling, and M. Katz, "Generalized planning in pddl domains with pretrained large language models," arXiv preprint arXiv:2305.11014, 2023. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Chen, Y., Arkin, J., Dawson, C., Zhang, Y., Roy, N., & Fan, C.
Affiliations
  • Massachusetts Institute of Technology (US)
  • IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
6695-6702
ISSN
1050-4729 (ISSN-L)
DOI
10.48550/arxiv.2306.06531
arXiv
2306.06531
OpenAlex
W4401414087
Cited by
146
References
0
Citations by year
2026: 43 · 2025: 54 · 2024: 43 · 2023: 6
FWCI
74.73
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University) (public-domain)
Funders
  • Office of Naval Research (N00014-22, W911NF-17-2-0181, N00014)
  • Army Research Laboratory (W911NF‐17‐2‐0181, W911NF)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
task and motion planning, large language models, natural language task descriptions, human-robot interaction, error detection and correction, time constraints

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review