CoChat

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.

VeriPlan: Integrating Formal Verification and LLMs into End-User Planning

Open access
Lee, Christine P., Porfirio, David, Wang, Xinyu Jessica, Zhao, Kevin Chenkai, Mutlu, Bilge · 2025 · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

Abstract

Automated planning is traditionally the domain of experts, utilized in fields like manufacturing and healthcare with the aid of expert planning tools. Recent advancements in LLMs have made planning more accessible to everyday users due to their potential to assist users with complex planning tasks. However, LLMs face several application challenges within end-user planning, including consistency, accuracy, and user trust issues. This paper introduces VeriPlan, a system that applies formal verification techniques, specifically model checking, to enhance the reliability and flexibility of LLMs for end-user planning. In addition to the LLM planner, VeriPlan includes three additional core features -- a rule translator, flexibility sliders, and a model checker -- that engage users in the verification process. Through a user study (n=12), we evaluate VeriPlan, demonstrating improvements in the perceived quality, usability, and user satisfaction of LLMs. Our work shows the effective integration of formal verification and user-control features with LLMs for end-user planning tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3706598.3714113

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 73 references

Research works printed but not linked: 39

  1. Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:https://arXiv.org/abs/2303.08774 (2023). · arXiv
  2. Chi-Min Chan Weize Chen Yusheng Su Jianxuan Yu Wei Xue Shanghang Zhang Jie Fu and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:https://arXiv.org/abs/2308.07201 (2023). · arXiv
  3. Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:https://arXiv.org/abs/2304.05128 (2023). · arXiv
  4. Yanan Chen Ali Pesaranghader Tanmana Sadhu and Dong Hoon Yi. 2024. Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let’s Take TravelPlanner as an Example. arXiv preprint arXiv:https://arXiv.org/abs/2408.06318 (2024). · arXiv
  5. Zhibin Gou Zhihong Shao Yeyun Gong Yelong Shen Yujiu Yang Nan Duan and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:https://arXiv.org/abs/2305.11738 (2023). · arXiv
  6. Atharva Gundawar Mudit Verma Lin Guan Karthik Valmeekam Siddhant Bhambri and Subbarao Kambhampati. 2024. Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning. arxiv:https://arXiv.org/abs/2405.20625 [cs.AI] https://arxiv.org/abs/2405.20625 · arXiv
  7. Christian Hensel Sebastian Junges Joost-Pieter Katoen Tim Quatmann and Matthias Volk. 2022. The probabilistic model checker Storm. International Journal on Software Tools for Technology Transfer (2022) 1–22.
  8. Hui Huang Yingqi Qu Jing Liu Muyun Yang and Tiejun Zhao. 2024. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv preprint arXiv:https://arXiv.org/abs/2403.02839 (2024). · arXiv
  9. Shiyuan Huang Siddarth Mamidanna Shreedhar Jangam Yilun Zhou and Leilani H Gilpin. 2023. Can large language models explain themselves? a study of llm-generated self-explanations. arXiv preprint arXiv:https://arXiv.org/abs/2310.11207 (2023). · arXiv
  10. Xu Huang Weiwen Liu Xiaolong Chen Xingmei Wang Hao Wang Defu Lian Yasheng Wang Ruiming Tang and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. arXiv preprint arXiv:https://arXiv.org/abs/2402.02716 (2024). · arXiv
  11. Zhenlan Ji Daoyuan Wu Pingchuan Ma Zongjie Li and Shuai Wang. 2024. Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs. arXiv preprint arXiv:https://arXiv.org/abs/2404.17833 (2024). · arXiv
  12. Yucheng Jin, Bruno De Lemos Ribeiro Pinto Cardoso, and Katrien Verbert. 2017. How do different levels of user control affect cognitive load and acceptance of recommendations?. In IntRS@ RecSys. 35–42.
  13. Subbarao Kambhampati Karthik Valmeekam Lin Guan Kaya Stechly Mudit Verma Siddhant Bhambri Lucas Saldyt and Anil Murthy. 2024. LLMs Can’t Plan But Can Help Planning in LLM-Modulo Frameworks. arXiv preprint arXiv:https://arXiv.org/abs/2402.01817 (2024). · arXiv
  14. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. 2024. Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks. In Proceedings of the 41st International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 22895–22907. https://proceedings.mlr.press/v235/kambhampati24a.html
  15. Hannah Rose Kirk Andrew M Bean Bertie Vidgen Paul Röttger and Scott A Hale. 2023. The past present and better future of feedback learning in large language models for subjective human preferences and values. arXiv preprint arXiv:https://arXiv.org/abs/2310.07629 (2023). · arXiv
  16. Ryan Koo Minhwa Lee Vipul Raheja Jong Inn Park Zae Myung Kim and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:https://arXiv.org/abs/2309.17012 (2023). · arXiv
  17. Shiyang Li Jianshu Chen Yelong Shen Zhiyu Chen Xinlu Zhang Zekun Li Hong Wang Jing Qian Baolin Peng Yi Mao et al. 2022. Explanations from large language models make small reasoners better. arXiv preprint arXiv:https://arXiv.org/abs/2210.06726 (2022). · arXiv
  18. Zhenwen Liang Ye Liu Tong Niu Xiangliang Zhang Yingbo Zhou and Semih Yavuz. 2024. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:https://arXiv.org/abs/2410.05318 (2024). · arXiv
  19. Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. 2023. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learning. PMLR, 1084–1110.
  20. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 43447–43478. https://proceedings.neurips.cc/paper_files/paper/2023/file/871ed095b734818cfba48db6aeb25a62-Paper-Conference.pdf
  21. Ana Marasović Iz Beltagy Doug Downey and Matthew E Peters. 2021. Few-shot self-rationalization with natural language prompts. arXiv preprint arXiv:https://arXiv.org/abs/2111.08284 (2021). · arXiv
  22. Anne-Sophie Mayer Franz Strich and Marina Fiedler. 2020. Unintended Consequences of Introducing AI Systems for Decision Making. MIS Quarterly Executive 19 4 (2020).
  23. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning. PMLR, 26106–26128.
  24. Lin Ning Luyang Liu Jiaxing Wu Neo Wu Devora Berlowitz Sushant Prakash Bradley Green Shawn O’Banion and Jun Xie. 2024. User-LLM: Efficient LLM Contextualization with User Embeddings. arXiv preprint arXiv:https://arXiv.org/abs/2402.13598 (2024). · arXiv
  25. Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare , Vol. 11. MDPI, 887.
  26. 10.1109/ICCV51070.2023.00280 · DOI · Paper page
  27. Lukas Teufelberger Xintong Liu Zhipeng Li Max Moebus and Christian Holz. 2024. LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows. arXiv preprint arXiv:https://arXiv.org/abs/2407.21593 (2024). · arXiv
  28. Miles Turpin Julian Michael Ethan Perez and Samuel Bowman. 2024. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36 (2024).
  29. Karthik Valmeekam Matthew Marquez Sarath Sreedharan and Subbarao Kambhampati. 2023. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems 36 (2023) 75993–76005.
  30. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.
  31. Helena Vasconcelos Gagan Bansal Adam Fourney Q Vera Liao and Jennifer Wortman Vaughan. 2023. Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in AI-powered code completions. arXiv preprint arXiv:https://arXiv.org/abs/2302.07248 (2023). · arXiv
  32. 10.1145/3613904.3641960 · DOI
  33. Sarah Wiegreffe Jack Hessel Swabha Swayamdipta Mark Riedl and Yejin Choi. 2021. Reframing human-AI collaboration for generating free-text explanations. arXiv preprint arXiv:https://arXiv.org/abs/2112.08674 (2021). · arXiv
  34. Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2024. MathChat: Converse to Tackle Challenging Math Problems with LLM Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  35. Jia-Yu Yao Kun-Peng Ning Zhen-Hui Liu Mu-Nan Ning Yu-Yang Liu and Li Yuan. 2023. Llm lies: Hallucinations are not bugs but features as adversarial examples. arXiv preprint arXiv:https://arXiv.org/abs/2310.01469 (2023). · arXiv
  36. Bin Zhang Hangyu Mao Jingqing Ruan Ying Wen Yang Li Shao Zhang Zhiwei Xu Dapeng Li Ziyue Li Rui Zhao et al. 2023. Controlling large language model-based agents for large-scale decision-making: An actor-critic approach. arXiv preprint arXiv:https://arXiv.org/abs/2311.13884 (2023). · arXiv
  37. Tao Zhang Yanjun Shen Wenjing Luo Yan Zhang Hao Liang Fan Yang Mingan Lin Yujing Qiao Weipeng Chen Bin Cui et al. 2024. Cfbench: A comprehensive constraints-following benchmark for llms. arXiv preprint arXiv:https://arXiv.org/abs/2408.01122 (2024). · arXiv
  38. Xinghua Zhang Bowen Yu Haiyang Yu Yangyu Lv Tingwen Liu Fei Huang Hongbo Xu and Yongbin Li. 2023. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:https://arXiv.org/abs/2308.01862 (2023). · arXiv
  39. Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2024).

Cited by

See all 22

Metadata

Authors
Lee, C. P., Porfirio, D., Wang, X. J., Zhao, K. C., & Mutlu, B.
Affiliations
  • University of Wisconsin–Madison (US)
  • United States Naval Research Laboratory (US)
Venue
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2025-04-25
Pages
1-19
OpenAlex
W4409779994
Cited by
21
References
73
Citations by year
2026: 13 · 2025: 8
FWCI
20.02
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (1925043)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
automated planning, large language models, formal verification, model checking, rule translation, user control, user trust, usability, user satisfaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: VeriPlan: Integrating Formal Verification and LLMs into End-User Planning. https://app.cochat.ai/papers/10.1145/3706598.3714113
The paper
Lee, C. P., Porfirio, D., Wang, X. J., Zhao, K. C., & Mutlu, B. (2025). VeriPlan: Integrating Formal Verification and LLMs into End-User Planning. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3714113

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.