The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
VeriPlan: Integrating Formal Verification and LLMs into End-User Planning
Open accessAbstract
Automated planning is traditionally the domain of experts, utilized in fields like manufacturing and healthcare with the aid of expert planning tools. Recent advancements in LLMs have made planning more accessible to everyday users due to their potential to assist users with complex planning tasks. However, LLMs face several application challenges within end-user planning, including consistency, accuracy, and user trust issues. This paper introduces VeriPlan, a system that applies formal verification techniques, specifically model checking, to enhance the reliability and flexibility of LLMs for end-user planning. In addition to the LLM planner, VeriPlan includes three additional core features -- a rule translator, flexibility sliders, and a model checker -- that engage users in the verification process. Through a user study (n=12), we evaluate VeriPlan, demonstrating improvements in the perceived quality, usability, and user satisfaction of LLMs. Our work shows the effective integration of formal verification and user-control features with LLMs for end-user planning tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work shows improved trust in users when they participate in verification.”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“VeriPlan [16] lets end-users approve logical rules (in natural language) and choose which to enforce as hard or soft constraints.”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“LLMs are prone to generating text that appears structurally coherent but contains factual inaccuracies or nonsensical information, a phenomenon known as hallucination [6, 32, 68, 83].”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
(2008) · Cited by 4,811
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803 · cited 1×
“…for decision-making or plan generation such as the healthcare domain ( e.g., ensuring diagnosis or treatment plans adhere to medical protocols [27]) or financial underwriting ( e.g., creating financial plans while complying with regulatory rules [67]), sliders can emphasize under-adhered-to…”
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
PRISM 4.0: Verification of Probabilistic Real-Time Systems
Kwiatkowska, M., Norman, G., & Parker, D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-22110-1_47 · Cited by 2,361 · cited 2×
“Before being presented to the user, each translation is converted back into natural language by the PRISM and LTL translator for user readability (step e ).”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Self-citationDesigning Web Usability: The Practice of Simplicity
Nielsen, J. (1999) · Cited by 1,778
PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains
Fox, M., & Long, D. (2003) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1129 · Cited by 1,741 · cited 2×
“Given the complexity and critical nature of these tasks, entire research communities and industries have dedicated themselves to building and utilizing planning tools ( e.g., [18, 38, 73]) that support fore-sight, decision-making, and the intricate coordination required for effective outcomes.”
Research works printed but not linked: 39
- Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:https://arXiv.org/abs/2303.08774 (2023). · arXiv
- Chi-Min Chan Weize Chen Yusheng Su Jianxuan Yu Wei Xue Shanghang Zhang Jie Fu and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:https://arXiv.org/abs/2308.07201 (2023). · arXiv
- Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:https://arXiv.org/abs/2304.05128 (2023). · arXiv
- Yanan Chen Ali Pesaranghader Tanmana Sadhu and Dong Hoon Yi. 2024. Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let’s Take TravelPlanner as an Example. arXiv preprint arXiv:https://arXiv.org/abs/2408.06318 (2024). · arXiv
- Zhibin Gou Zhihong Shao Yeyun Gong Yelong Shen Yujiu Yang Nan Duan and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:https://arXiv.org/abs/2305.11738 (2023). · arXiv
- Atharva Gundawar Mudit Verma Lin Guan Karthik Valmeekam Siddhant Bhambri and Subbarao Kambhampati. 2024. Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning. arxiv:https://arXiv.org/abs/2405.20625 [cs.AI] https://arxiv.org/abs/2405.20625 · arXiv
- Christian Hensel Sebastian Junges Joost-Pieter Katoen Tim Quatmann and Matthias Volk. 2022. The probabilistic model checker Storm. International Journal on Software Tools for Technology Transfer (2022) 1–22.
- Hui Huang Yingqi Qu Jing Liu Muyun Yang and Tiejun Zhao. 2024. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv preprint arXiv:https://arXiv.org/abs/2403.02839 (2024). · arXiv
- Shiyuan Huang Siddarth Mamidanna Shreedhar Jangam Yilun Zhou and Leilani H Gilpin. 2023. Can large language models explain themselves? a study of llm-generated self-explanations. arXiv preprint arXiv:https://arXiv.org/abs/2310.11207 (2023). · arXiv
- Xu Huang Weiwen Liu Xiaolong Chen Xingmei Wang Hao Wang Defu Lian Yasheng Wang Ruiming Tang and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. arXiv preprint arXiv:https://arXiv.org/abs/2402.02716 (2024). · arXiv
- Zhenlan Ji Daoyuan Wu Pingchuan Ma Zongjie Li and Shuai Wang. 2024. Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs. arXiv preprint arXiv:https://arXiv.org/abs/2404.17833 (2024). · arXiv
- Yucheng Jin, Bruno De Lemos Ribeiro Pinto Cardoso, and Katrien Verbert. 2017. How do different levels of user control affect cognitive load and acceptance of recommendations?. In IntRS@ RecSys. 35–42.
- Subbarao Kambhampati Karthik Valmeekam Lin Guan Kaya Stechly Mudit Verma Siddhant Bhambri Lucas Saldyt and Anil Murthy. 2024. LLMs Can’t Plan But Can Help Planning in LLM-Modulo Frameworks. arXiv preprint arXiv:https://arXiv.org/abs/2402.01817 (2024). · arXiv
- Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. 2024. Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks. In Proceedings of the 41st International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 22895–22907. https://proceedings.mlr.press/v235/kambhampati24a.html
- Hannah Rose Kirk Andrew M Bean Bertie Vidgen Paul Röttger and Scott A Hale. 2023. The past present and better future of feedback learning in large language models for subjective human preferences and values. arXiv preprint arXiv:https://arXiv.org/abs/2310.07629 (2023). · arXiv
- Ryan Koo Minhwa Lee Vipul Raheja Jong Inn Park Zae Myung Kim and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:https://arXiv.org/abs/2309.17012 (2023). · arXiv
- Shiyang Li Jianshu Chen Yelong Shen Zhiyu Chen Xinlu Zhang Zekun Li Hong Wang Jing Qian Baolin Peng Yi Mao et al. 2022. Explanations from large language models make small reasoners better. arXiv preprint arXiv:https://arXiv.org/abs/2210.06726 (2022). · arXiv
- Zhenwen Liang Ye Liu Tong Niu Xiangliang Zhang Yingbo Zhou and Semih Yavuz. 2024. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:https://arXiv.org/abs/2410.05318 (2024). · arXiv
- Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. 2023. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learning. PMLR, 1084–1110.
- Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 43447–43478. https://proceedings.neurips.cc/paper_files/paper/2023/file/871ed095b734818cfba48db6aeb25a62-Paper-Conference.pdf
- Ana Marasović Iz Beltagy Doug Downey and Matthew E Peters. 2021. Few-shot self-rationalization with natural language prompts. arXiv preprint arXiv:https://arXiv.org/abs/2111.08284 (2021). · arXiv
- Anne-Sophie Mayer Franz Strich and Marina Fiedler. 2020. Unintended Consequences of Introducing AI Systems for Decision Making. MIS Quarterly Executive 19 4 (2020).
- Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning. PMLR, 26106–26128.
- Lin Ning Luyang Liu Jiaxing Wu Neo Wu Devora Berlowitz Sushant Prakash Bradley Green Shawn O’Banion and Jun Xie. 2024. User-LLM: Efficient LLM Contextualization with User Embeddings. arXiv preprint arXiv:https://arXiv.org/abs/2402.13598 (2024). · arXiv
- Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare , Vol. 11. MDPI, 887.
- 10.1109/ICCV51070.2023.00280 · DOI · Paper page
- Lukas Teufelberger Xintong Liu Zhipeng Li Max Moebus and Christian Holz. 2024. LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows. arXiv preprint arXiv:https://arXiv.org/abs/2407.21593 (2024). · arXiv
- Miles Turpin Julian Michael Ethan Perez and Samuel Bowman. 2024. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36 (2024).
- Karthik Valmeekam Matthew Marquez Sarath Sreedharan and Subbarao Kambhampati. 2023. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems 36 (2023) 75993–76005.
- Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.
- Helena Vasconcelos Gagan Bansal Adam Fourney Q Vera Liao and Jennifer Wortman Vaughan. 2023. Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in AI-powered code completions. arXiv preprint arXiv:https://arXiv.org/abs/2302.07248 (2023). · arXiv
- 10.1145/3613904.3641960 · DOI
- Sarah Wiegreffe Jack Hessel Swabha Swayamdipta Mark Riedl and Yejin Choi. 2021. Reframing human-AI collaboration for generating free-text explanations. arXiv preprint arXiv:https://arXiv.org/abs/2112.08674 (2021). · arXiv
- Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2024. MathChat: Converse to Tackle Challenging Math Problems with LLM Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
- Jia-Yu Yao Kun-Peng Ning Zhen-Hui Liu Mu-Nan Ning Yu-Yang Liu and Li Yuan. 2023. Llm lies: Hallucinations are not bugs but features as adversarial examples. arXiv preprint arXiv:https://arXiv.org/abs/2310.01469 (2023). · arXiv
- Bin Zhang Hangyu Mao Jingqing Ruan Ying Wen Yang Li Shao Zhang Zhiwei Xu Dapeng Li Ziyue Li Rui Zhao et al. 2023. Controlling large language model-based agents for large-scale decision-making: An actor-critic approach. arXiv preprint arXiv:https://arXiv.org/abs/2311.13884 (2023). · arXiv
- Tao Zhang Yanjun Shen Wenjing Luo Yan Zhang Hao Liang Fan Yang Mingan Lin Yujing Qiao Weipeng Chen Bin Cui et al. 2024. Cfbench: A comprehensive constraints-following benchmark for llms. arXiv preprint arXiv:https://arXiv.org/abs/2408.01122 (2024). · arXiv
- Xinghua Zhang Bowen Yu Haiyang Yu Yangyu Lv Tingwen Liu Fei Huang Hongbo Xu and Yongbin Li. 2023. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:https://arXiv.org/abs/2308.01862 (2023). · arXiv
- Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2024).
Cited by
A systematic review of ethical considerations of large language models in healthcare and medicine
Fareed, M., Fatima, M., Uddin, J., Ahmed, A., & Sattar, M. A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1653631 · Cited by 72
Navigating Embodied Intelligence: Enabling Technologies, Security and Privacy, and Emerging Trends
Wang, Y., Niu, X., Ba, J., Su, Z., & Du, L. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3649049 · Cited by 20
“Additionally, Lee et al. [109] introduce VeriPlan, a human-in-the-loop planning framework that combines LLMs with formal verification via model checking to improve the reliability and usability of LLM-generated plans.”
AgentGuard: Runtime Verification of AI Agents
Koohestani, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00023 · Cited by 9
“Similarly, VeriPlan applies the probabilistic model checker PRISM to verify that an LLM-generated plan for an end user satisfies a set of user-defined constraints [17].”
Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification
Mitchell, J., & Shaaban, Y. (2025) · Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages · doi:10.1145/3759425.3763390 · Cited by 8
“VeriPlan [16] lets end-users approve logical rules (in natural language) and choose which to enforce as hard or soft constraints.”
Huynh, L., & McNamara, D. S. (2025) · Applied Sciences · doi:10.3390/app152212128 · Cited by 13
Metadata
- Authors
- Lee, C. P., Porfirio, D., Wang, X. J., Zhao, K. C., & Mutlu, B.
- Affiliations
- University of Wisconsin–Madison (US)
- United States Naval Research Laboratory (US)
- Venue
- Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2025-04-25
- Pages
- 1-19
- OpenAlex
- W4409779994
- Cited by
- 21
- References
- 73
- Citations by year
- 2026: 13 · 2025: 8
- FWCI
- 20.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1925043)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- automated planning, large language models, formal verification, model checking, rule translation, user control, user trust, usability, user satisfaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: VeriPlan: Integrating Formal Verification and LLMs into End-User Planning. https://app.cochat.ai/papers/10.1145/3706598.3714113
- The paper
- Lee, C. P., Porfirio, D., Wang, X. J., Zhao, K. C., & Mutlu, B. (2025). VeriPlan: Integrating Formal Verification and LLMs into End-User Planning. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3714113
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.