Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification
Open accessPublished version of a preprint
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 33 of the 52 sources its claims cite were checked for support.
Abstract
“Vibe coding” — the practice of developing software through iteratively conversing with a large language model (LLM) — has exploded in popularity within the last year. However, developers report key limitations including the accumulation of technical debt, security issues, and code churn to achieve satisfactory results. We argue that these pitfalls result from LLMs’ inability to reconcile accumulating human-imposed constraints during vibe coding, with developers inadvertently failing to resolve contradictions because LLMs prioritize user commands over code consistency. Given LLMs’ receptiveness to verification-based feedback, we argue that formal methods can mitigate these pitfalls, making vibe coding more reliable. However, we posit that integrating formal methods must transcend existing approaches that combine formal methods and LLMs. We advocate for a side-car system throughout the vibe coding process which: (1) Autoformalizes specifications (2) Validates against targets, (3) Delivers actionable feedback to the LLM, and (4) Allows intuitive developer influence on specifications.
Overview
CoChat Summary
TL;DR
This position paper argues that vibe coding’s accumulating constraints cause technical debt and security issues because LLMs cannot reconcile them, and proposes “Vibe Reasoning” (Type III systems): a developer-first side-car that autoformalizes specifications, continuously verifies critical invariants, and feeds actionable results back to the LLM. It sketches a TypeScript proof-of-concept integrated with Claude Code and calls for formal methods research on developer-centric verification.
Key findings
- The paper argues vibe coding pitfalls stem from LLMs prioritizing user commands over code consistency, leading to constraint-reconciliation decay; it cites a repository-level evaluation of 318 benchmark programs where the strongest model generates secure and correct code only 28% of the time.
- Existing LLM-plus-formal-methods work is classified into Type I systems (formal methods filter solutions and provide feedback) and Type II systems (formal methods post-process LLM outputs by construction); both rely on humans to supply formal methods components, motivating the proposed Type III.
- The proposed Type III side-car has four parts: autoformalize specifications, validate against targets, deliver actionable feedback to the LLM, and allow intuitive developer influence on specifications.
- A TypeScript proof-of-concept implements four autoformalization templates—Exhaustive Switch, Discriminated Union, Union Alias, and satisfies Guard—integrated with Claude Code via hooks; the authors report it can flag and fix both bugs in the toy Listing 1 example.
Methods
- Position paper, not an empirical study: it synthesizes reported challenges and prior work, proposes a taxonomy of Type I/II/III systems, and describes an architecture.
- Proof-of-concept implementation in TypeScript of the Type III side-car, integrated with Claude Code via hooks that run on code changes; autoformalization uses LLM-generated specifications from predefined templates, and verification currently uses lightweight syntactic checks and compilation success.
- Evaluation is illustrative only: a toy TypeScript example in Listing 1, with results described as promising and no quantitative metrics reported.
Limitations
- The paper does not state a formal limitations section.
- Evident from the text: the proof-of-concept is small and TypeScript-specific; verification is currently limited to lightweight syntactic checks and compilation success, and future implementations will use more sophisticated code provenance for patterns that cannot be syntactically verified.
- Evident from the text: there is no quantitative evaluation or user study; the human-collaboration mechanisms are proposed but not tested.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
16 Valid · 1 Not valid · 11 Not checked · 17 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“This results in constraintreconciliation decay : as sessions lengthen and dependencies accumulate, success rates fall.”
What the sources say
- Supportedcites [44] [44] Zhenyu Zhang, Tianyu Zheng, Linguo Luan, Linyuan Wang, Chao Shen, Qian Li, and Yang Liu. 2025. CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation. CoRR abs/2504.21751 (2025). arXiv:2504.21751 doi: 10.48550/ARXIV.2504.21751 - source
Section 4.2 (pp. 7–8) reports that Pass@1 declines sharply as turn counts rise and that models struggle more as dependency structures become complex. The paper does not use the term “constraint-reconciliation decay,” but the claimed performance pattern matches its findings. (LLM, conf=0.92)
- Disputedcites [10] [10] Vu Le Hai, Myoungkyu Song, Minsuk Choi, Jae-Young Lee, Hyungho Choi, and Hakjoo Oh. 2024. RepoExec: Evaluate Code Generation with a Repository-Level Executable Benchmark. CoRR abs/2406.11927 (2024). arXiv:2406.11927 doi: 10.48550/ARXIV.2406.11927 - sourceExperimental — accuracy not yet measured
Table 5 in §6.1 reports that Pass@1 rose across three debugging rounds for GPT-3.5 and WizardCoder and remained unchanged for CodeLlama-13b-Python; it did not fall as rounds accumulated. The multi-round debugging discussion also reports improved dependency use. The paper does not establish a separate trend for sessions in which dependencies accumulate. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
- Could not verifycites [22] [22] Zhenyu Liu, Zhaoyu Li, Yuan Huang, Xinming Wei, Qi Lu, Fan Yang, Xiaoxing Ma, and Zhongyi Lu. 2024. FullStack Bench: Evaluating LLMs as Full Stack Coders. CoRR abs/2412.00535 (2024). arXiv:2412.00535 doi: 10.48550/ARXIV.2412.00535 - sourceExperimental — accuracy not yet measured
The Implementation Details section evaluates each problem as a separate user prompt, and §4.5 compares performance by problem difficulty. Neither passage addresses whether success rates fall as sessions lengthen or dependencies accumulate; searches for session length, sequential dependencies, and constraint reconciliation found no such analysis. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Repository-level evaluations reveal similar cross-file integration and dependency breaks [10, 22].”
What the sources say
- Partly supportedcites [10] [10] Vu Le Hai, Myoungkyu Song, Minsuk Choi, Jae-Young Lee, Hyungho Choi, and Hakjoo Oh. 2024. RepoExec: Evaluate Code Generation with a Repository-Level Executable Benchmark. CoRR abs/2406.11927 (2024). arXiv:2406.11927 doi: 10.48550/ARXIV.2406.11927 - sourceExperimental — accuracy not yet measured
The Introduction and Section 3 describe repository-level evaluation of functional correctness and dependency use; the dependency-extraction section explicitly includes cross-file dependencies. The Additional information section gives examples of models failing to use available dependencies, but our copy does not establish distinct cross-file integration breaks. (LLM, conf=0.82)
- Could not verifycites [22] [22] Zhenyu Liu, Zhaoyu Li, Yuan Huang, Xinming Wei, Qi Lu, Fan Yang, Xiaoxing Ma, and Zhongyi Lu. 2024. FullStack Bench: Evaluating LLMs as Full Stack Coders. CoRR abs/2412.00535 (2024). arXiv:2412.00535 doi: 10.48550/ARXIV.2412.00535 - sourceExperimental — accuracy not yet measured
The Code Benchmark section mentions repository-level benchmarks, but does not report that they reveal cross-file integration or dependency breaks. Searches for those failure types found only discussion of sandbox execution and package-version compatibility, not the claimed evaluation finding. The cited passages provide no page numbers. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many existing works combine LLMs and formal methods, highlighting the ability of LLMs to effectively incorporate feedback from a verification engine.”
What the sources say
- Partly supportedcites [2] [2] Saikat Chakraborty, Shuvendu K. Lahiri, Sarah Fakhoury, Akash Lal, Madanlal Musuvathi, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, and Nikhil Swamy. 2023. Ranking LLM-Generated Loop Invariants for Program Verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , HoudaBouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 9164-9175. doi: 10.18653/V1/2023.FINDINGSEMNLP.614 - sourceExperimental — accuracy not yet measured
The Introduction describes using LLMs to synthesize loop invariants for formal verification, supporting the general combination of LLMs and formal methods. But Related Work attributes the incorporation of verifier feedback to other machine-learning techniques and says this paper’s LLM-based approach does not require a verifier at inference time. It does not establish that LLMs effectively incorporate such feedback. (LLM, conf=0.88)
- Supportedcites [12] [12] Adharsh Kamath, Aditya Senthilnathan, Saikat Chakraborty, Pantazis Deligiannis, Shuvendu K. Lahiri, Akash Lal, Aseem Rastogi, Subhajit Roy, and Rahul Sharma. 2023. Finding Inductive Loop Invariants using Large Language Models. CoRR abs/2311.07948 (2023). arXiv:2311.07948 doi: 10.48550/ARXIV.2311.07948 - source
Section 3.3 (Repair; page unavailable) describes giving verification-oracle errors to an LLM to revise loop invariants. Section 4.4 (RQ4; page unavailable) reports that this feedback-based repair verified 15 additional benchmarks compared with the version without repair, supporting the claim that LLMs can use verification feedback effectively. The paper also situates its combination of LLMs and formal verification among related work in Section 5. (LLM, conf=0.87)
- Partly supportedcites [20] [20] Chang Liu, Xiwei Wu, Yuan Feng, Qinxiang Cao, and Junchi Yan. 2023. Towards General Loop Invariant Generation via Coordinating Symbolic Execution and Large Language Models. CoRR abs/2311.10483 (2023). arXiv:2311.10483 doi: 10.48550/ARXIV.2311.10483 - sourceExperimental — accuracy not yet measured
Section 4.2 describes this paper’s interactive framework, in which verification tools check and refine LLM-generated terms, and the updated assertions are used in subsequent inference. The paper thus supports an example of LLM–formal-methods collaboration, but it does not establish that many existing works demonstrate LLMs effectively incorporating verification-engine feedback. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Multi-turn coding benchmarks show that performance steeply declines compared to single-turn tasks, dominated by failures to globalize changes across helpers and state [44].”
What the sources say
- Partly supportedcites [44] [44] Zhenyu Zhang, Tianyu Zheng, Linguo Luan, Linyuan Wang, Chao Shen, Qian Li, and Yang Liu. 2025. CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation. CoRR abs/2504.21751 (2025). arXiv:2504.21751 doi: 10.48550/ARXIV.2504.21751 - sourceExperimental — accuracy not yet measured
The Main Experiments (p. 6) report a substantial drop in performance from single-turn to multi-turn tasks. The error analysis (pp. 8–9) identifies failures involving global context, shared state, and helper-function integration, but does not say these dominate across the benchmark: CodeFlowBench-Comp is dominated by incomplete reasoning, while CodeFlowBench-Repo places more emphasis on globalization and instruction misinterpretation. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Success of agentic systems on static benchmarks (SWE-bench Verified) do not carry over to dynamic, evolving tasks (SWEbench-Live) [24, 43].”
What the sources say
- Partly supportedcites [24] [24] Matias Martinez and Xavier Franch. 2025. Dissecting the SWEBench Leaderboards: Profiling Submitters and Architectures of LLMand Agent-Based Repair Systems. CoRR abs/2506.17208 (2025). arXiv:2506.17208 doi: 10.48550/arXiv.2506.17208 - sourceExperimental — accuracy not yet measured
Section 3.2 (p. 30) reports competitive results for autonomous single-agent systems on SWE-Bench Verified. Section 4.4 (p. 42) describes SWE-bench-Live as continuously updated with recent data, but does not compare agent performance on the two benchmarks or show that success on Verified fails to carry over to Live. (LLM, conf=0.82)
- Supportedcites [43] [43] Peiyuan Zhang, Carlos E. Jimenez, Zhiyuan (Brett) Chen, Dedy K. T. Suhendi, Hiroki Nakamura, David D. L. Liu, Yingfei Xie, Shrikanth S. Narayanan, John P. DiCarlo, and Sujay V. Khandekar. 2025. SWEbench Goes Live! CoRR abs/2505.23419 (2025). arXiv:2505.23419 doi: 10.48550/ARXIV.2505.23419 - source
Section 4.2 (p. 7) reports that the same OpenHands–Claude 3.7 Sonnet setup resolved 43.20% of SWE-bench Verified tasks but scored less than half as well on SWE-bench-Live. The Introduction (p. 2) likewise says performance was significantly lower on the live benchmark, supporting the claim that success on the static benchmark did not carry over at the same level. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
13 more claims · 4 with their cited sources checked
Create a free account to see all 45 claims and every source verdict.
- Probably not a substantial claim. 22
- It may state two separate claims as one. 10
Citations
Cites
Why don't software developers use static analysis tools to find bugs?
Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606613 · Cited by 568
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 493
Lessons from building static analysis tools at Google
Sadowski, C., Aftandilian, E., Eagle, A., Miller-Cushon, L., & Jaspan, C. (2018) · Communications of the ACM · doi:10.1145/3188720 · Cited by 242 · cited 1×
“Furthermore, even prior to vibe coding, industry reports show that it is difficult to get developers to adopt static analysis in their workflows [5, 11, 30, 34].”
Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
Wu, J., Zhu, J., Liu, Y., Xu, M., & Jin, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1383 · Cited by 227
Scaling static analyses at Facebook
Distefano, D., Fähndrich, M., Logozzo, F., & O'Hearn, P. W. (2019) · Communications of the ACM · doi:10.1145/3338112 · Cited by 212 · cited 1×
“Furthermore, even prior to vibe coding, industry reports show that it is difficult to get developers to adopt static analysis in their workflows [5, 11, 30, 34].”
Wei, Y., Xia, C. S., & Zhang, L. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616271 · Cited by 115
Type-Constrained Code Generation with Language Models
Mündler, N., He, J., Wang, H., Sen, K., Song, D. X., & Vechev, M. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729274 · Cited by 83 · cited 1×
“Type II systems (Figure 2, right) use formal methods to post-process LLM outputs so that it satisfies specifications by construction (e.g., constrained decoding [25, 28, 29], program completion [35]).”
Ranking LLM-Generated Loop Invariants for Program Verification
Chakraborty, S., Lahiri, S., Fakhoury, S., Lal, A., Musuvathi, M., Rastogi, A., Senthilnathan, A., Sharma, R., & Swamy, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.614 · Cited by 37
Towards General Loop Invariant Generation: A Benchmark of Programs with Memory Manipulation
Liu, C., Wu, X., Feng, Y., Cao, Q., & Yan, J. (2024) · neural information processing systems · doi:10.52202/079017-4101 · Cited by 35
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.19443 · Cited by 32
Research works printed but not linked: 3
- Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. 2024. Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview.net/forum?id=Th8JPEmH4z
- Zenan Li, Yifan Wu, Zhaoyu Li, Xinming Wei, Xian Zhang, Fan Yang, and Xiaoxing Ma. 2024. Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). http://papers.nips.cc/paper_files/paper/2024/hash/6034a661584af6c28fd97a6f23e56c0a-Abstract-Conference.html
- D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems 28, Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). Curran Associates, Inc., 2503–2511. https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
Cited by
Responsible Vibe Coding: Architecture, Opportunities, and Research Agenda
Elgendy, I. A., Dwivedi, Y. K., Al-Sharafi, M. A., Hosny, M., Helal, M. Y. I., Crick, T., Hughes, L., Alwahaishi, S. S., Mahmud, M., Dutot, V., et al. (2026) · Journal of Computer Information Systems · doi:10.1080/08874417.2026.2621186 · Cited by 16
Hope or Hype? Understanding Vibe Coding through Software Practitioner Discussions
Meem, F. N., Zohra, F. T., Smith, J., & Johnson, B. (2026) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3794860.3794904 · Cited by 1
AXL: Executable Intent for Governed LLM-Based Software Maintenance
Vella, S., Sharieh, S., Ferworn, A., & Sharieh, M. (2026) · Conference proceedings - Canadian Conference on Electrical and Computer Engineering · doi:10.1109/ccece68150.2026.11610535
“Mitchell and Shaaban propose autoformalizing developer intent into specifications that can be verified and then use a sidecar architecture to check LLM outputs against the specifications rather than relying solely on text generation [11].”
Vibe Coding in Software Development: A Multivocal Literature ReviewPreprint
Siddeeq, S., Waseem, M., Kemell, K., Saari, M., Rasku, J., & Abrahamsson, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7194448 · Cited by 2
Vella, S., Sharieh, S., MalekSharieh, & Ferworn, A. (2026) · International Conference on Federated Learning and Intelligent Computing Systems (FLICS) · doi:10.1109/flics70075.2026.11621935
“This is a viewpoint thatemphasizess that vibe coding, which is open-ended, must transition to vise coding where formal verification and structured feedback guide the model to a valid state [24].”
Metadata
- Authors
- Mitchell, J., & Shaaban, Y.
- Affiliations
- University of Southern California (US)
- Workato, Palo Alto, USA
- Venue
- Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages
- Publisher
- ACM
- Published
- 2025-10-09
- Pages
- 84-90
- OpenAlex
- W4415002704
- Cited by
- 8
- References
- 35
- Citations by year
- 2026: 8
- FWCI
- 9.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM SIGPLAN International Workshop on Language Models and Programming Languages (LMPL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
- Keywords
- vibe coding, large language models, formal verification, formal methods, LLM-assisted software development, technical debt
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification. https://app.cochat.ai/papers/10.1145/3759425.3763390
- The paper
- Mitchell, J., & Shaaban, Y. (2025). Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification. Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages. https://doi.org/10.1145/3759425.3763390
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.