VeriPlan: Integrating Formal Verification and LLMs into End-User Planning
Open accessThe DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index.
Abstract
Automated planning is traditionally the domain of experts, utilized in fields like manufacturing and healthcare with the aid of expert planning tools. Recent advancements in LLMs have made planning more accessible to everyday users due to their potential to assist users with complex planning tasks. However, LLMs face several application challenges within end-user planning, including consistency, accuracy, and user trust issues. This paper introduces VeriPlan, a system that applies formal verification techniques, specifically model checking, to enhance the reliability and flexibility of LLMs for end-user planning. In addition to the LLM planner, VeriPlan includes three additional core features -- a rule translator, flexibility sliders, and a model checker -- that engage users in the verification process. Through a user study (n=12), we evaluate VeriPlan, demonstrating improvements in the perceived quality, usability, and user satisfaction of LLMs. Our work shows the effective integration of formal verification and user-control features with LLMs for end-user planning tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work shows improved trust in users when they participate in verification.”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“VeriPlan [16] lets end-users approve logical rules (in natural language) and choose which to enforce as hard or soft constraints.”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“LLMs are prone to generating text that appears structurally coherent but contains factual inaccuracies or nonsensical information, a phenomenon known as hallucination [6, 32, 68, 83].”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.03629 · Cited by 4,981
(2008) · Cited by 4,811
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803 · cited 1×
“…for decision-making or plan generation such as the healthcare domain ( e.g., ensuring diagnosis or treatment plans adhere to medical protocols [27]) or financial underwriting ( e.g., creating financial plans while complying with regulatory rules [67]), sliders can emphasize under-adhered-to…”
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
PRISM 4.0: Verification of Probabilistic Real-Time Systems
Kwiatkowska, M., Norman, G., & Parker, D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-22110-1_47 · Cited by 2,361 · cited 2×
“Before being presented to the user, each translation is converted back into natural language by the PRISM and LTL translator for user readability (step e ).”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Self-citationDesigning Web Usability: The Practice of Simplicity
Nielsen, J. (1999) · Cited by 1,778
PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains
Fox, M., & Long, D. (2003) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1129 · Cited by 1,741 · cited 2×
“Given the complexity and critical nature of these tasks, entire research communities and industries have dedicated themselves to building and utilizing planning tools ( e.g., [18, 38, 73]) that support fore-sight, decision-making, and the intricate coordination required for effective outcomes.”
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1×
“These issues are especially concerning in safety or mission-critical domains, such as healthcare or military applications, where reliance on incorrect outputs can have severe consequences [42, 50, 84].”
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548 · cited 1×
“The user is hosting a dinner party on Wednesday at 6:00 PM with multiple guests, requiring the preparation of various dishes to accommodate different dietary preferences, such as vegetarian and gluten-free.”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274 · cited 1×
“Studies highlight the difficulty users face in formulating prompts that elicit desired responses [39, 60, 99, 102, 120].”
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 2×
“These issues can further lead to risks of users over-relying on LLM-generated outputs without sufficient critical evaluation, underscoring the need for mechanisms that support users’ safe and reliable interactions with LLMs [32, 68].”
MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K., Qiao, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_10 · Cited by 929
Clarke, V., & Braun, V. (2014) · Encyclopedia of Critical Psychology · doi:10.1007/978-1-4614-5583-7_311 · Cited by 885
Designing the user interface: Strategies for effective human-computer interaction
Sriram, D. (1987) · Artificial Intelligence in Engineering · doi:10.1016/0954-1810(87)90189-0 · Cited by 756
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644 · cited 1×
“The lack of interpretability further complicates users’ safe reliability, as users often struggle to understand the reasoning behind the output of the LLM [60, 66, 71, 116, 125].”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644 · cited 1×
“The lack of interpretability further complicates users’ safe reliability, as users often struggle to understand the reasoning behind the output of the LLM [60, 66, 71, 116, 125].”
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1×
“Looking ahead, the integration of formal verification processes can play an increasingly critical role as LLMs and AI systems are increasingly used for planning contexts [98, 115, 119].”
Simonson, I. (2005) · Journal of Marketing · doi:10.1509/jmkg.69.1.32.55512 · Cited by 523 · cited 1×
“This stage can also be particularly effective for user engagement, as users might find it easier to define high-level preferences compared to specifying granular operational details, or they may not always have detailed insights or clarity into their needs and preferences [48, 95].”
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493 · cited 4×
“Additionally, the cognitive demands placed on users—such as monitoring and deciding on strategies for prompting and interaction— exacerbate these challenges [99, 102].”
InfluentialGhallab, M., Nau, D., & Traverso, P. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781139583923 · Cited by 439 · cited 1×
“Automated planning refers to automated techniques that decide what an agent does, namely the steps that it takes to achieve a goal, rather than how it performs each step [19].”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Song, C. H., Sadler, B. M., Wu, J., Chao, W.-L., Washington, C., & Su, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00280 · Cited by 435 · cited 1×
“Given a natural language prompt or set of prompts, LLMs are demonstrably capable planners [62, 94, 97] without requiring the user to directly interact with low-level planning languages or libraries.”
Large Language Models: A Survey
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., & Gao, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06196 · Cited by 325
Generalized Planning in PDDL Domains with Pretrained Large Language Models
Silver, T., Dan, S., Srinivas, K., Tenenbaum, J. B., Kaelbling, L. P., & Katz, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.30006 · Cited by 305
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300 · cited 1×
“Another usability barrier is users’ difficulty understanding how prompts influence outputs and building accurate mental models of the system’s behavior and the reasoning behind it [7, 101, 107].”
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 1×
“Similarly, interactive environments with user-controllable parameters enable experimentation, helping users build a better understanding of LLM capabilities [34, 61, 64, 100].”
Concise finite-domain representations for PDDL planning tasks
Helmert, M. (2009) · Artificial Intelligence · doi:10.1016/j.artint.2008.10.013 · Cited by 229 · cited 1×
“The technical interfaces often lack intuitiveness, providing rigid workflows and low-level feedback [25, 79, 89].”
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219 · cited 1×
“Another usability barrier is users’ difficulty understanding how prompts influence outputs and building accurate mental models of the system’s behavior and the reasoning behind it [7, 101, 107].”
Ebbinghaus, H.-D., Flum, J., & Thomas, W. (1994) · Springer eBooks · doi:10.1007/978-1-4757-2355-7 · Cited by 215
Mathews, S. M. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-22868-2_90 · Cited by 214 · cited 1×
“The lack of interpretability further complicates users’ safe reliability, as users often struggle to understand the reasoning behind the output of the LLM [60, 66, 71, 116, 125].”
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2024) · arXiv · doi:10.48550/arxiv.2404.00971 · Cited by 188
OffsetBias: Leveraging Debiased Data for Tuning Evaluators
Park, J., Jwa, S., Meiying, R., Kim, D., & Choi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.57 · Cited by 177
Synthesis for Robots: Guarantees and Feedback for Robot Behavior
Kress-Gazit, H., Lahijanian, M., & Raman, V. (2018) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-060117-104838 · Cited by 169 · cited 1×
“Automated planning —the search for sequences of actions that guide an autonomous agent from an initial state to a goal state [44]— has traditionally been the domain of experts.”
Applications, Challenges, and Future Directions of Human-in-the-Loop Learning
Kumar, S., Datta, S., Singh, V., Datta, D., Kumar Singh, S., & Sharma, R. (2024) · IEEE Access · doi:10.1109/access.2024.3401547 · Cited by 168 · cited 1×
“Existing research shows that user input on preferences and domain knowledge during initial interactions is important to effectively guide system behavior [46].”
Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R., Pondoc, C., Agrawala, M., & Seifert, C. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 162 · cited 3×
“Additionally, the cognitive demands placed on users—such as monitoring and deciding on strategies for prompting and interaction— exacerbate these challenges [99, 102].”
Understanding Large-Language Model (LLM)-powered Human-Robot Interaction
Kim, C. Y., Lee, C. P., & Mutlu, B. (2024) · Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3610977.3634966 · Cited by 162
Self-citationSensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
Suh, S., Min, B., Palani, S., & Xia, H. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606756 · Cited by 151
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A., & Arawjo, I. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676450 · Cited by 151 · cited 1×
“Additionally, other approaches involve humans in evaluating and correcting outputs [91, 110].”
Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
Suh, S., Min, B., Palani, S., & Xia, H. (2023) · arXiv · doi:10.48550/arxiv.2305.11483 · Cited by 151
Large Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.511 · Cited by 138
Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
Jiang, P., Rayan, J., Dow, S. P., & Xia, H. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606737 · Cited by 135 · cited 3×
“Similarly, interactive environments with user-controllable parameters enable experimentation, helping users build a better understanding of LLM capabilities [34, 61, 64, 100].”
A Survey of Hallucination in Large Foundation Models
Rawte, V., Sheth, A., & Das, A. (2023) · arXiv · doi:10.48550/arxiv.2309.05922 · Cited by 133 · cited 1×
“LLMs are prone to generating text that appears structurally coherent but contains factual inaccuracies or nonsensical information, a phenomenon known as hallucination [6, 32, 68, 83].”
A synthesis of automated planning and reinforcement learning for efficient, robust decision-making
Leonetti, M., Iocchi, L., & Stone, P. (2016) · Artificial Intelligence · doi:10.1016/j.artint.2016.07.004 · Cited by 130 · cited 1×
“Planning has been applied in professional settings, including production planning in manufacturing, medical resource planning in healthcare, project planning in construction, and route and fleet planning in transportation [8, 52, 53].”
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Wang, X., Kim, H., Rahman, S., Mitra, K., & Miao, Z. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641960 · Cited by 128 · cited 2×
“Additionally, other approaches involve humans in evaluating and correcting outputs [91, 110].”
Self-citationPallagani, V., Muppasani, B. C., Roy, K., Fabiano, F., Loreggia, A., Murugesan, K., Srivastava, B., Rossi, F., Horesh, L., & Sheth, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31503 · Cited by 125 · cited 1×
“Existing work has shown that, despite the increasing attention LLMs are receiving as planning tools, they are insufficient for planning and self-verification, particularly in the planning domain [30, 36, 77, 105, 106, 111].”
Khurana, A., Subramonyam, H., & Chilana, P. K. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645200 · Cited by 114 · cited 2×
“Recent research has also highlighted several challenges, including difficulties with prompt input and navigation, limitations of text-only interfaces, and issues with evaluating LLMs’ consistency and accuracy in meeting user needs [39, 85, 99, 102].”
Putting Users in Control of their Recommendations
Harper, F. M., Xu, F., Kaur, H., Condiff, K., Chang, S., & Terveen, L. (2015) · Proceedings of the 9th ACM Conference on Recommender Systems · doi:10.1145/2792838.2800179 · Cited by 108 · cited 1×
“User control has long been recognized as a critical feature in human-computer interaction, influencing user experience and outcome quality [3, 22, 23, 75, 93].”
Moodplay: Interactive Mood-based Music Discovery and Recommendation
Andjelkovic, I., Parra, D., & O’Donovan, J. (2016) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/2930238.2930280 · Cited by 89
Harambam, J., Bountouridis, D., Makhortykh, M., & van Hoboken, J. (2019) · Proceedings of the 13th ACM Conference on Recommender Systems · doi:10.1145/3298689.3347014 · Cited by 84 · cited 1×
“This stage can also be particularly effective for user engagement, as users might find it easier to define high-level preferences compared to specifying granular operational details, or they may not always have detailed insights or clarity into their needs and preferences [48, 95].”
EvaluLLM: LLM assisted evaluation of generative outputs
Desmond, M. J., Ashktorab, Z., Pan, Q., Dugan, C., & Johnson, J. M. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640544.3645216 · Cited by 80 · cited 1×
“One notable direction includes using LLMs for evaluation [14, 124, 127] or orchestrating multi-agent systems to verify outputs [9, 24, 56, 72].”
Authoring and Verifying Human-Robot Interactions
Porfirio, D. J., Sauppé, A., Albarghouthi, A., & Mutlu, B. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242634 · Cited by 79
Self-citationHuang, H., Bu, X., Zhou, H., Qu, Y., Liu, J., Yang, M., Xu, B., & Zhao, T. (2024) · arXiv · doi:10.48550/arxiv.2403.02839 · Cited by 78
A PDDL Based Tool for Automatic Web Service Composition
Peer, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30122-6_11 · Cited by 74 · cited 1×
“The technical interfaces often lack intuitiveness, providing rigid workflows and low-level feedback [25, 79, 89].”
"We Need Structured Output": Towards User-centered Constraints on Large Language Model Output
Liu, M. X., Liu, F., Fiannaca, A. J., Koo, T., Dixon, L., Terry, M., & Cai, C. J. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650756 · Cited by 73
Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses
Ma, X., Mishra, S. R., Liu, A., Su, S. Y., Chen, J., Kulkarni, C. E., Cheng, H.-T., Le, Q. V., & Ed, H. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651093 · Cited by 73 · cited 4×
“Similarly, interactive environments with user-controllable parameters enable experimentation, helping users build a better understanding of LLM capabilities [34, 61, 64, 100].”
InfluentialMaking Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · cited 1×
“These methods have been applied to complex tasks such as mathematical reasoning [55, 114, 122], semantic reasoning [10, 57, 74], and data annotation [110].”
Shang, F., Zhao, F., Zhang, M., Sun, U., & Shi, J. (2024) · International Journal of Innovative Research in Engineering & Management · doi:10.55524/ijirem.2024.11.4.6 · Cited by 58 · cited 1×
“As LLMs become increasingly used as tools for beyond planning contexts ( e.g., personalization [76, 90]), verification methods can enable users to safely and effectively guide, collaborate with, and customize these systems to meet their specific needs.”
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55 · cited 1×
“Ultimately, this interplay between user input and system capabilities can foster a more effective and user-centered verification process.”
Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., & Liang, Y. (2023) · arXiv · doi:10.48550/arxiv.2302.01560 · Cited by 55 · cited 1×
“Existing work has shown that, despite the increasing attention LLMs are receiving as planning tools, they are insufficient for planning and self-verification, particularly in the planning domain [30, 36, 77, 105, 106, 111].”
Speak From Heart: An Emotion-Guided LLM-Based Multimodal Method for Emotional Dialogue Generation
Liu, C., Xie, Z., Zhao, S., Zhou, J., Xu, T., Li, M., & Chen, E. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658104 · Cited by 52 · cited 1×
“These methods have been applied to complex tasks such as mathematical reasoning [55, 114, 122], semantic reasoning [10, 57, 74], and data annotation [110].”
Sarkar, A. (2023) · Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work · doi:10.1145/3596671.3597650 · Cited by 50 · cited 1×
“Recent research has also highlighted several challenges, including difficulties with prompt input and navigation, limitations of text-only interfaces, and issues with evaluating LLMs’ consistency and accuracy in meeting user needs [39, 85, 99, 102].”
Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents
Yang, Z., Raman, S. S., Shah, A., & Tellex, S. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611447 · Cited by 49 · cited 1×
“Finally, a growing area of research incorporates constraint-based approaches, such as applying constraints to planning in robotics [117], creating datasets with constraints for evaluation [123], or generating plans that adhere to multiple constraints [115].”
Large Language Models as General Pattern Machines
Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., & Zeng, A. (2023) · arXiv · doi:10.48550/arxiv.2307.04721 · Cited by 49 · cited 1×
“The lack of interpretability further complicates users’ safe reliability, as users often struggle to understand the reasoning behind the output of the LLM [60, 66, 71, 116, 125].”
Leiser, F., Eckhardt, S., Knaeble, M., Maedche, A., Schwabe, G., & Sunyaev, A. (2023) · Mensch und Computer 2023 · doi:10.1145/3603555.3603565 · Cited by 44 · cited 1×
“Finally, LLMs are prone to “hallucinations”—coherent but incorrect information—that undermine user trust, usability, and satisfaction [51].”
LLM-Guided Formal Verification Coupled with Mutation Testing
Hassan, M., Ahmadi-Pour, S., Qayyum, K., Jha, C. K., & Drechsler, R. (2024) · 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE) · doi:10.23919/date58400.2024.10546729 · Cited by 43 · cited 1×
“One notable direction includes using LLMs for evaluation [14, 124, 127] or orchestrating multi-agent systems to verify outputs [9, 24, 56, 72].”
Koga, S. (2023) · Pathology International · doi:10.1111/pin.13382 · Cited by 41 · cited 1×
“These issues are especially concerning in safety or mission-critical domains, such as healthcare or military applications, where reliance on incorrect outputs can have severe consequences [42, 50, 84].”
Statistical Power of Within and Between-Subjects Designs in Economic ExperimentsPreprint
Bellemare, C., Bissonnette, L., & Kröger, S. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2529895 · Cited by 39
“The Fridge Door is Open”–Temporal Verification of a Robotic Assistant’s Behaviours
Dixon, C., Webster, M., Saunders, J., Fisher, M., & Dautenhahn, K. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10401-0_9 · Cited by 26 · cited 1×
“Linear Temporal Logic (LTL) is a commonly used representation to express requirements, or properties , in domains such as assistive robotics [15] and autonomous navigation [59].”
Post Hoc Explanations of Language Models Can Improve Language ModelsPreprint
Krishna, S., Ma, J., Slack, D., Ghandeharioun, A., Singh, S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3006112/v1 · Cited by 25
Transforming Robot Programs Based on Social Context
Porfirio, D. J., Sauppé, A., Albarghouthi, A., & Mutlu, B. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376355 · Cited by 23 · cited 1×
“Moreover, users must invest significant effort in creating detailed system models, specifying states, transitions, and probabilities [81, 82, 86]—tasks that demand technical expertise and are highly time-consuming.”
Self-citationThe AI-DEC: A Card-based Design Method for User-centered AI Explanations
Lee, C. P., Lee, M. K., & Mutlu, B. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661576 · Cited by 22 · cited 1×
“Below, we discuss how our findings address the research questions and present design implications for integrating verification processes and user control features into future systems.”
Self-citationMostajabdaveh, M., Yu, T. T., Ramamonjison, R., Carenini, G., Zhou, Z., & Zhang, Y. (2024) · INFOR Information Systems and Operational Research · doi:10.1080/03155986.2024.2381306 · Cited by 22 · cited 1×
“One notable direction includes using LLMs for evaluation [14, 124, 127] or orchestrating multi-agent systems to verify outputs [9, 24, 56, 72].”
TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.01622 · Cited by 22 · cited 3×
“Looking ahead, the integration of formal verification processes can play an increasingly critical role as LLMs and AI systems are increasingly used for planning contexts [98, 115, 119].”
REX: Designing User-centered Repair and Explanations to Address Robot Failures
Lee, C. P., Praveena, P., & Mutlu, B. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661559 · Cited by 21 · cited 1×
“…the effective use of LLMs in end-user planning, LLM-based planning systems must not only be designed to be reliable , but the user-LLM interaction must also be designed to support correction when the system produces incorrect or unacceptable output—a core principle of human-centered AI [2, 49].”
Self-citationRecent Advances and Future Challenges in Automated Manufacturing Planning
Bourne, D., Corney, J., & Gupta, S. K. (2011) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.3593411 · Cited by 20 · cited 1×
“Planning has been applied in professional settings, including production planning in manufacturing, medical resource planning in healthcare, project planning in construction, and route and fleet planning in transportation [8, 52, 53].”
Goal-Oriented End-User Programming of Robots
Porfirio, D. J., Roberts, M., & Hiatt, L. M. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610977.3634974 · Cited by 20 · cited 1×
“Although planning tools are typically intended for expert users, recent work has engaged novice users in the planning process through visualization [13] and plan creation [80].”
Self-citationAuthr: A Task Authoring Environment for Human-Robot Teams
Schoen, A. J., Henrichs, C., Strohkirch, M., & Mutlu, B. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415872 · Cited by 19 · cited 1×
“However, these planning tools pose significant challenges for end-users due to their reliance on complex formal languages and abstract logic formulas [16, 31, 88], which are difficult to learn and apply.”
Self-citationYour Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models
Schellaert, W., Martínez-Plumed, F., Vold, K., Burden, J., A. M. Casares, P., Sheng Loe, B., Reichart, R., Ó hÉigeartaigh, S., Korhonen, A., & Hernández-Orallo, J. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14157 · Cited by 19 · cited 2×
“Below, we discuss how our findings address the research questions and present design implications for integrating verification processes and user control features into future systems.”
Fang, Z., Qi, J., Fan, L., Huang, J., Jin, Y., & Yang, T. (2022) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2022.101853 · Cited by 18
Programming assistance based on contracts and modular verification in the automation domain
Hurnaus, D., & Prähofer, H. (2010) · ACM Symposium on Applied Computing · doi:10.1145/1774088.1774614 · Cited by 9 · cited 1×
“However, these planning tools pose significant challenges for end-users due to their reliance on complex formal languages and abstract logic formulas [16, 31, 88], which are difficult to learn and apply.”
Koga, S. (2023) · medRxiv · doi:10.1101/2023.08.03.23293401 · Cited by 5
Structure Synthesis for Extended Robot State Automata
Sauer, L., & Henrich, D. (2022) · Mechanisms and machine science · doi:10.1007/978-3-031-04870-8_9 · Cited by 4
Planning Domain Simulation: An Interactive System for Plan Visualisation
Pellegrin, E. D., & Petrick, R. P. A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31469 · Cited by 3 · cited 1×
“Although planning tools are typically intended for expert users, recent work has engaged novice users in the planning process through visualization [13] and plan creation [80].”
LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
Son, G., Ko, H., Lee, H., Kim, Y., & Hong, S. (2024) · arXiv · doi:10.48550/arxiv.2409.11239 · Cited by 1
Research works printed but not linked: 38
- Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:https://arXiv.org/abs/2303.08774 (2023). · arXiv
- Chi-Min Chan Weize Chen Yusheng Su Jianxuan Yu Wei Xue Shanghang Zhang Jie Fu and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:https://arXiv.org/abs/2308.07201 (2023). · arXiv
- Xinyun Chen Maxwell Lin Nathanael Schärli and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:https://arXiv.org/abs/2304.05128 (2023). · arXiv
- Yanan Chen Ali Pesaranghader Tanmana Sadhu and Dong Hoon Yi. 2024. Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let’s Take TravelPlanner as an Example. arXiv preprint arXiv:https://arXiv.org/abs/2408.06318 (2024). · arXiv
- Zhibin Gou Zhihong Shao Yeyun Gong Yelong Shen Yujiu Yang Nan Duan and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:https://arXiv.org/abs/2305.11738 (2023). · arXiv
- Atharva Gundawar Mudit Verma Lin Guan Karthik Valmeekam Siddhant Bhambri and Subbarao Kambhampati. 2024. Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning. arxiv:https://arXiv.org/abs/2405.20625 [cs.AI] https://arxiv.org/abs/2405.20625 · arXiv
- Christian Hensel Sebastian Junges Joost-Pieter Katoen Tim Quatmann and Matthias Volk. 2022. The probabilistic model checker Storm. International Journal on Software Tools for Technology Transfer (2022) 1–22.
- Shiyuan Huang Siddarth Mamidanna Shreedhar Jangam Yilun Zhou and Leilani H Gilpin. 2023. Can large language models explain themselves? a study of llm-generated self-explanations. arXiv preprint arXiv:https://arXiv.org/abs/2310.11207 (2023). · arXiv
- Xu Huang Weiwen Liu Xiaolong Chen Xingmei Wang Hao Wang Defu Lian Yasheng Wang Ruiming Tang and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. arXiv preprint arXiv:https://arXiv.org/abs/2402.02716 (2024). · arXiv
- Zhenlan Ji Daoyuan Wu Pingchuan Ma Zongjie Li and Shuai Wang. 2024. Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs. arXiv preprint arXiv:https://arXiv.org/abs/2404.17833 (2024). · arXiv
- Yucheng Jin, Bruno De Lemos Ribeiro Pinto Cardoso, and Katrien Verbert. 2017. How do different levels of user control affect cognitive load and acceptance of recommendations?. In IntRS@ RecSys. 35–42.
- Subbarao Kambhampati Karthik Valmeekam Lin Guan Kaya Stechly Mudit Verma Siddhant Bhambri Lucas Saldyt and Anil Murthy. 2024. LLMs Can’t Plan But Can Help Planning in LLM-Modulo Frameworks. arXiv preprint arXiv:https://arXiv.org/abs/2402.01817 (2024). · arXiv
- Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. 2024. Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks. In Proceedings of the 41st International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 22895–22907. https://proceedings.mlr.press/v235/kambhampati24a.html
- Hannah Rose Kirk Andrew M Bean Bertie Vidgen Paul Röttger and Scott A Hale. 2023. The past present and better future of feedback learning in large language models for subjective human preferences and values. arXiv preprint arXiv:https://arXiv.org/abs/2310.07629 (2023). · arXiv
- Ryan Koo Minhwa Lee Vipul Raheja Jong Inn Park Zae Myung Kim and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:https://arXiv.org/abs/2309.17012 (2023). · arXiv
- Shiyang Li Jianshu Chen Yelong Shen Zhiyu Chen Xinlu Zhang Zekun Li Hong Wang Jing Qian Baolin Peng Yi Mao et al. 2022. Explanations from large language models make small reasoners better. arXiv preprint arXiv:https://arXiv.org/abs/2210.06726 (2022). · arXiv
- Zhenwen Liang Ye Liu Tong Niu Xiangliang Zhang Yingbo Zhou and Semih Yavuz. 2024. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:https://arXiv.org/abs/2410.05318 (2024). · arXiv
- Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. 2023. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learning. PMLR, 1084–1110.
- Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 43447–43478. https://proceedings.neurips.cc/paper_files/paper/2023/file/871ed095b734818cfba48db6aeb25a62-Paper-Conference.pdf
- Ana Marasović Iz Beltagy Doug Downey and Matthew E Peters. 2021. Few-shot self-rationalization with natural language prompts. arXiv preprint arXiv:https://arXiv.org/abs/2111.08284 (2021). · arXiv
- Anne-Sophie Mayer Franz Strich and Marina Fiedler. 2020. Unintended Consequences of Introducing AI Systems for Decision Making. MIS Quarterly Executive 19 4 (2020).
- Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning. PMLR, 26106–26128.
- Lin Ning Luyang Liu Jiaxing Wu Neo Wu Devora Berlowitz Sushant Prakash Bradley Green Shawn O’Banion and Jun Xie. 2024. User-LLM: Efficient LLM Contextualization with User Embeddings. arXiv preprint arXiv:https://arXiv.org/abs/2402.13598 (2024). · arXiv
- Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare , Vol. 11. MDPI, 887.
- 10.1109/ICCV51070.2023.00280 · DOI · Paper page
- Lukas Teufelberger Xintong Liu Zhipeng Li Max Moebus and Christian Holz. 2024. LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows. arXiv preprint arXiv:https://arXiv.org/abs/2407.21593 (2024). · arXiv
- Miles Turpin Julian Michael Ethan Perez and Samuel Bowman. 2024. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36 (2024).
- Karthik Valmeekam Matthew Marquez Sarath Sreedharan and Subbarao Kambhampati. 2023. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems 36 (2023) 75993–76005.
- Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.
- Helena Vasconcelos Gagan Bansal Adam Fourney Q Vera Liao and Jennifer Wortman Vaughan. 2023. Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in AI-powered code completions. arXiv preprint arXiv:https://arXiv.org/abs/2302.07248 (2023). · arXiv
- 10.1145/3613904.3641960 · DOI · Paper page
- Sarah Wiegreffe Jack Hessel Swabha Swayamdipta Mark Riedl and Yejin Choi. 2021. Reframing human-AI collaboration for generating free-text explanations. arXiv preprint arXiv:https://arXiv.org/abs/2112.08674 (2021). · arXiv
- Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2024. MathChat: Converse to Tackle Challenging Math Problems with LLM Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
- Jia-Yu Yao Kun-Peng Ning Zhen-Hui Liu Mu-Nan Ning Yu-Yang Liu and Li Yuan. 2023. Llm lies: Hallucinations are not bugs but features as adversarial examples. arXiv preprint arXiv:https://arXiv.org/abs/2310.01469 (2023). · arXiv
- Bin Zhang Hangyu Mao Jingqing Ruan Ying Wen Yang Li Shao Zhang Zhiwei Xu Dapeng Li Ziyue Li Rui Zhao et al. 2023. Controlling large language model-based agents for large-scale decision-making: An actor-critic approach. arXiv preprint arXiv:https://arXiv.org/abs/2311.13884 (2023). · arXiv
- Tao Zhang Yanjun Shen Wenjing Luo Yan Zhang Hao Liang Fan Yang Mingan Lin Yujing Qiao Weipeng Chen Bin Cui et al. 2024. Cfbench: A comprehensive constraints-following benchmark for llms. arXiv preprint arXiv:https://arXiv.org/abs/2408.01122 (2024). · arXiv
- Xinghua Zhang Bowen Yu Haiyang Yu Yangyu Lv Tingwen Liu Fei Huang Hongbo Xu and Yongbin Li. 2023. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:https://arXiv.org/abs/2308.01862 (2023). · arXiv
- Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2024).
Cited by
A systematic review of ethical considerations of large language models in healthcare and medicine
Fareed, M., Fatima, M., Uddin, J., Ahmed, A., & Sattar, M. A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1653631 · Cited by 73
Navigating Embodied Intelligence: Enabling Technologies, Security and Privacy, and Emerging Trends
Wang, Y., Niu, X., Ba, J., Su, Z., & Du, L. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3649049 · Cited by 20
“Additionally, Lee et al. [109] introduce VeriPlan, a human-in-the-loop planning framework that combines LLMs with formal verification via model checking to improve the reliability and usability of LLM-generated plans.”
AgentGuard: Runtime Verification of AI Agents
Koohestani, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00023 · Cited by 31
“Similarly, VeriPlan applies the probabilistic model checker PRISM to verify that an LLM-generated plan for an end user satisfies a set of user-defined constraints [17].”
Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification
Mitchell, J., & Shaaban, Y. (2025) · Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages · doi:10.1145/3759425.3763390 · Cited by 8
“VeriPlan [16] lets end-users approve logical rules (in natural language) and choose which to enforce as hard or soft constraints.”
Huynh, L., & McNamara, D. S. (2025) · Applied Sciences · doi:10.3390/app152212128 · Cited by 13
Metadata
- Authors
- Lee, C. P., Porfirio, D., Wang, X. J., Zhao, K. C., & Mutlu, B.
- Affiliations
- University of Wisconsin–Madison (US)
- United States Naval Research Laboratory (US)
- Venue
- Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2025-04-25
- Pages
- 1-19
- OpenAlex
- W4409779994
- Cited by
- 21
- References
- 73
- Citations by year
- 2026: 13 · 2025: 8
- FWCI
- 20.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1925043)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- automated planning, large language models, formal verification, model checking, rule translation, user control, user trust, usability, user satisfaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.