The DOI matches its DataCite record.
Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning
Abstract
Large language models (LLMs) have revolutionized a large variety of NLP tasks. An active debate is to what extent they can do reasoning and planning. Prior work has assessed the latter in the specific context of PDDL planning, based on manually converting three PDDL domains into natural language (NL) prompts. Here we automate this conversion step, showing how to leverage an LLM to automatically generate NL prompts from PDDL input. Our automatically generated NL prompts result in similar LLM-planning performance as the previous manually generated ones. Beyond this, the automation enables us to run much larger experiments, providing for the first time a broad evaluation of LLM planning performance in PDDL. Our NL prompts yield better performance than PDDL prompts and simple template-based NL prompts. Compared to symbolic planners, LLM planning lags far behind; but in some domains, our best LLM configuration scales up further than A$^\star$ using LM-cut.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Results show that even SOTA models struggle with long-horizon planning.”
From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Stein et al., 2023
In the citing paper
“AutoPlanBench (Stein et al., 2023) focuses on evaluating planning in everyday scenarios, demonstrating that even SoTA LLM agents lag behind classical symbolic planners.”
From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Stein et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 28
- Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y.; Cortes, O.; David, B.; Finn, C.; Fu, C.; Gopalakrishnan, K.; Hausman, K.; Herzog, A.; Ho, D.; Hsu, J.; Ibarz, J.; Ichter, B.; Irpan, A.; Jang, E.; Ruano, R. J.; Jeffrey, K.; Jesmonth, S.; Joshi, N. J.; Julian, R.; Kalashnikov, D.; Kuang, Y.; Lee, K.-H.; Levine, S.; Lu, Y.; Luu, L.; Parada, C.; Pastor, P.; Quiambao, J.; Rao, K.; Rettinghouse, J.; Reyes, D.; Sermanet, P.; Siev- ers, N.; Tan, C.; Toshev, A.; Vanhoucke, V.; Xia, F.; Xiao, T.; Xu, P.; Xu, S.; Yan, M.; and Zeng, A. 2022. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. In 6th Conference on Robot Learning (CoRL).
- Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brock- man, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray, S.; Ryder, N.; Pavlov, M.; Power, A.; Kaiser, L.; Bavarian, M.; Winter, C.; Tillet, P.; Such, F. P.; Cummings, D.; Plappert, M.; Chantzis, F.; Barnes, E.; Herbert-Voss, A.; Guss, W. H.; Nichol, A.; Paino, A.; Tezak, N.; Tang, J.; Babuschkin, I.; Balaji, S.; Jain, S.; Saunders, W.; Hesse, C.; Carr, A. N.; Leike, J.; Achiam, J.; Misra, V.; Morikawa, E.; Radford, A.; Knight, M.; Brundage, M.; Murati, M.; Mayer, K.; Welinder, P.; Mc- Grew, B.; Amodei, D.; McCandlish, S.; Sutskever, I.; and Zaremba, W. 2021. Evaluating Large Language Models Trained on Code. Preprint arXiv:2107.03374. · arXiv
- Eisenhut, J.; Schuler, X.; Fišer, D.; Höller, D.; Christakis, M.; and Hoffmann, J. 2024. New Fuzzing Biases for Action Policy Testing. Proceedings of the International Conference on Automated Planning and Scheduling, 34(1): 162-167.
- Gerevini, A.; Saetti, A.; and Serina, I. 2003. Planning through Stochastic Local Search and Temporal Action Graphs. Journal of Artificial Intelligence Research (JAIR), 20: 239-290.
- Gupta, T.; and Kembhavi, A. 2023. Visual Programming: Compositional Visual Reasoning Without Training. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14953-14962.
- Hao, S.; Gu, Y.; Ma, H.; Hong, J.; Wang, Z.; Wang, D.; and Hu, Z. 2023. Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Process- ing, 8154-8173.
- Hazra, R.; Zuidberg Dos Martires, P.; and De Raedt, L. 2024. SayCanPay: Heuristic Planning with Large Language Mod- els Using Learnable Domain Knowledge. Proceedings of the AAAI Conference on Artificial Intelligence, 38(18): 20123- 20133.
- Helmert, M. 2009. Concise finite-domain representations for PDDL planning tasks. Artificial Intelligence, 173(5-6): 503-535.
- Helmert, M.; and Domshlak, C. 2009. Landmarks, Critical Paths and Abstractions: What's the Difference Anyway? In Proceedings of the 19th International Conference on Auto- mated Planning and Scheduling, ICAPS. AAAI.
- Hoffmann, J.; and Nebel, B. 2001. The FF Planning System: Fast Plan Generation Through Heuristic Search. Journal of Artificial Intelligence Research, 14: 253-302.
- Katz, M.; Kokel, H.; Srinivas, K.; and Sohrabi, S. 2024. Thought of Search: Planning with Language Models Through The Lens of Efficiency. In Advances in Neural In- formation Processing Systems, volume 37, 138491-138568.
- Khot, T.; Khashabi, D.; Richardson, K.; Clark, P.; and Sab- harwal, A. 2021. Text Modular Networks: Learning to De- compose Tasks in the Language of Existing Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1264-1279.
- Lin, B. Y.; Fu, Y.; Yang, K.; Brahman, F.; Huang, S.; Bha- gavatula, C.; Ammanabrolu, P.; Choi, Y.; and Ren, X. 2023. SwiftSage: A Generative Agent with Fast and Slow Think- ing for Complex Interactive Tasks. In Advances in Neural Information Processing Systems, volume 36, 23813-23825.
- Liu, B.; Jiang, Y.; Zhang, X.; Liu, Q.; Zhang, S.; Biswas, J.; and Stone, P. 2023. LLM+P: Empowering Large Lan- guage Models with Optimal Planning Proficiency. Preprint arXiv:2304.11477. · arXiv
- Prasad, A.; Koller, A.; Hartmann, M.; Clark, P.; Sabharwal, A.; Bansal, M.; and Khot, T. 2024. ADaPT: As-Needed Decomposition and Planning with Language Models. In Findings of the Association for Computational Linguistics: NAACL 2024, 4226-4252.
- Qi, P.; Zhang, Y.; Zhang, Y.; Bolton, J.; and Manning, C. D. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
- Rossetti, N.; Tummolo, M.; Gerevini, A. E.; Putelli, L.; Se- rina, I.; Chiari, M.; and Olivato, M. 2024. Learning General Policies for Planning through GPT Models. Proceedings of the International Conference on Automated Planning and Scheduling, 34(1): 500-508.
- Shridhar, M.; Yuan, X.; Cote, M.-A.; Bisk, Y.; Trischler, A.; and Hausknecht, M. 2021. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Inter- national Conference on Learning Representations.
- Silver, T.; Dan, S.; Srinivas, K.; Tenenbaum, J. B.; Kael- bling, L. P.; and Katz, M. 2024. Generalized Planning in PDDL Domains with Pretrained Large Language Models. In 38th AAAI Conference on Artificial Intelligence (AAAI'24).
- Silver, T.; Hariprasad, V.; Shuttleworth, R. S.; Kumar, N.; Lozano-Pérez, T.; and Kaelbling, L. P. 2022. PDDL Plan- ning with Pretrained Large Language Models. In NeurIPS 2022 Foundation Models for Decision Making Workshop.
- Valmeekam, K.; Marquez, M.; Olmo, A.; Sreedharan, S.; and Kambhampati, S. 2023a. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Plan- ning and Reasoning about Change. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Valmeekam, K.; Marquez, M.; Sreedharan, S.; and Kamb- hampati, S. 2023b. On the Planning Abilities of Large Lan- guage Models -A Critical Investigation. In Advances in Neural Information Processing Systems, 75993-76005.
- Wang, R.; Jansen, P.; Côté, M.-A.; and Ammanabrolu, P. 2022. ScienceWorld: Is your Agent Smarter than a 5th Grader? In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, 11279- 11298.
- Wang, Z.; Cai, S.; Chen, G.; Liu, A.; Ma, X. S.; and Liang, Y. 2023. Describe, Explain, Plan and Select: Interactive Plan- ning with LLMs Enables Open-World Multi-Task Agents. In Advances in Neural Information Processing Systems, vol- ume 36, 34153-34189.
- Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q. V.; and Zhou, D. 2022. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Sys- tems, volume 35, 24824-24837.
- Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K. R.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations.
- Zhou, A.; Yan, K.; Shlapentokh-Rothman, M.; Wang, H.; and Wang, Y.-X. 2024. Language agent tree search unifies reasoning, acting, and planning in language models. In Pro- ceedings of the 41st International Conference on Machine Learning, ICML'24.
- Zhu, X.; Chen, Y.; Tian, H.; Tao, C.; Su, W.; Yang, C.; Huang, G.; Li, B.; Lu, L.; Wang, X.; Qiao, Y.; Zhang, Z.; and Dai, J. 2023. Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory. Preprint arXiv:2305.17144. · arXiv
Cited by
Survey on Evaluation of LLM-based Agents
Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., & Shmueli-Scheuer, M. (2025) · arXiv · doi:10.48550/arxiv.2503.16416 · Cited by 26
Metadata
- Authors
- Stein, K., Fišer, D., Hoffmann, J., & Koller, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2311.09830
- References
- 0
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning. https://app.cochat.ai/papers/10.48550/arxiv.2311.09830
- The paper
- Stein, K., Fišer, D., Hoffmann, J., & Koller, A. (2023). Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning. arXiv. https://doi.org/10.48550/arxiv.2311.09830
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.