CoChat

Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification

Open access
Mitchell, Jacqueline, Shaaban, Yasser · 2025 · Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages

Published version of a preprint

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 33 of the 52 sources its claims cite were checked for support.

Abstract

“Vibe coding” — the practice of developing software through iteratively conversing with a large language model (LLM) — has exploded in popularity within the last year. However, developers report key limitations including the accumulation of technical debt, security issues, and code churn to achieve satisfactory results. We argue that these pitfalls result from LLMs’ inability to reconcile accumulating human-imposed constraints during vibe coding, with developers inadvertently failing to resolve contradictions because LLMs prioritize user commands over code consistency. Given LLMs’ receptiveness to verification-based feedback, we argue that formal methods can mitigate these pitfalls, making vibe coding more reliable. However, we posit that integrating formal methods must transcend existing approaches that combine formal methods and LLMs. We advocate for a side-car system throughout the vibe coding process which: (1) Autoformalizes specifications (2) Validates against targets, (3) Delivers actionable feedback to the LLM, and (4) Allows intuitive developer influence on specifications.

Overview

CoChat Summary

TL;DR

This position paper argues that vibe coding’s accumulating constraints cause technical debt and security issues because LLMs cannot reconcile them, and proposes “Vibe Reasoning” (Type III systems): a developer-first side-car that autoformalizes specifications, continuously verifies critical invariants, and feeds actionable results back to the LLM. It sketches a TypeScript proof-of-concept integrated with Claude Code and calls for formal methods research on developer-centric verification.

Key findings

  • The paper argues vibe coding pitfalls stem from LLMs prioritizing user commands over code consistency, leading to constraint-reconciliation decay; it cites a repository-level evaluation of 318 benchmark programs where the strongest model generates secure and correct code only 28% of the time.
  • Existing LLM-plus-formal-methods work is classified into Type I systems (formal methods filter solutions and provide feedback) and Type II systems (formal methods post-process LLM outputs by construction); both rely on humans to supply formal methods components, motivating the proposed Type III.
  • The proposed Type III side-car has four parts: autoformalize specifications, validate against targets, deliver actionable feedback to the LLM, and allow intuitive developer influence on specifications.
  • A TypeScript proof-of-concept implements four autoformalization templates—Exhaustive Switch, Discriminated Union, Union Alias, and satisfies Guard—integrated with Claude Code via hooks; the authors report it can flag and fix both bugs in the toy Listing 1 example.

Methods

  • Position paper, not an empirical study: it synthesizes reported challenges and prior work, proposes a taxonomy of Type I/II/III systems, and describes an architecture.
  • Proof-of-concept implementation in TypeScript of the Type III side-car, integrated with Claude Code via hooks that run on code changes; autoformalization uses LLM-generated specifications from predefined templates, and verification currently uses lightweight syntactic checks and compilation success.
  • Evaluation is illustrative only: a toy TypeScript example in Listing 1, with results described as promising and no quantitative metrics reported.

Limitations

  • The paper does not state a formal limitations section.
  • Evident from the text: the proof-of-concept is small and TypeScript-specific; verification is currently limited to lightweight syntactic checks and compilation success, and future implementations will use more sophisticated code provenance for patterns that cannot be syntactically verified.
  • Evident from the text: there is no quantitative evaluation or user study; the human-collaboration mechanisms are proposed but not tested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3759425.3763390

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

16 Valid · 1 Not valid · 11 Not checked · 17 Own results and interpretations

13 more claims · 4 with their cited sources checked

Create a free account to see all 45 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 22
  • It may state two separate claims as one. 10

Create a free account to see these claims.

Citations

Cites

Show all 49 references

Research works printed but not linked: 3

  1. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. 2024. Position: LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview.net/forum?id=Th8JPEmH4z
  2. Zenan Li, Yifan Wu, Zhaoyu Li, Xinming Wei, Xian Zhang, Fan Yang, and Xiaoxing Ma. 2024. Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). http://papers.nips.cc/paper_files/paper/2024/hash/6034a661584af6c28fd97a6f23e56c0a-Abstract-Conference.html
  3. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems 28, Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). Curran Associates, Inc., 2503–2511. https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf

Cited by

See all 8

Metadata

Authors
Mitchell, J., & Shaaban, Y.
Affiliations
  • University of Southern California (US)
  • Workato, Palo Alto, USA
Venue
Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages
Publisher
ACM
Published
2025-10-09
Pages
84-90
OpenAlex
W4415002704
Cited by
8
References
35
Citations by year
2026: 8
FWCI
9.58
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM SIGPLAN International Workshop on Language Models and Programming Languages (LMPL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Logic, programming, and type systems
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
Keywords
vibe coding, large language models, formal verification, formal methods, LLM-assisted software development, technical debt

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification. https://app.cochat.ai/papers/10.1145/3759425.3763390
The paper
Mitchell, J., & Shaaban, Y. (2025). Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification. Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages. https://doi.org/10.1145/3759425.3763390

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.