Efficient Guided Generation for Large Language Models

“We are concerned with the problem of generating sequences of tokens from a large language model (LLM) [Vaswani et al., 2017, Radford et al., 2019] that conform to regular expressions or context-free grammars (CFGs).”

“The vocabularies, V, are composed of strings from a fixed alphabet [Sennrich et al., 2015] and N is often on the order of 104 or larger.”

“…output usable under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et…”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“Outlines [1] reformulated guided generation as FSM state transitions with a vocabulary index, extending to CFGs via lexer-aware scanning over parser states—the paper whose problem statement and formal register this work parallels, and an explicit inspiration for GRID’s design.”

“The environment validates every ID against the set of children just presented; invalid IDs are rejected, which realizes the constrained-decoding principle of restricting generation to a valid candidate set [19] at the level of tool arguments.”

“We use the recently developed Outlines framework to obtain structured output in a yes/no format [35].”

“But the benefit comes with a catch: In GQR, the model must return one of a few exact labels, which entails a structured-output and instruction-following problem [47, 42].”

“Email: sjttu@tqu.edu.vn To thoroughly address formatting vulnerabilities, the constrained decoding technique is applied directly at the inference layer using GBNF (GGML Backus-Naur Form) grammar sampling [21].”

“Similarly, advanced interaction frameworks like Outlines [28] or LMQL [31] also use regular expressions and FSM mechanisms to coerce output shapes.”

“Beyond prompting, a more direct approach to output control is logit masking, i.e. forcing forbidden token probabilities to −∞ at every decoding step, as implemented in frameworks such as that proposed by Willard and Louf [15] and its predecessors Hokamp and Liu [16].”

“Guidance [12] and Out-lines [11] enforce structural validity during token sampling by restricting generation to valid JSON or other schemas.”

“Constrained decoding [20]–[22] underlies the FSM/JSON masking used for argument generation.”

“Willard and Louf [14] demonstrated efficient implementations of grammar-constrained decoding with little latency overhead.”

“While this approach ensures valid output formats, existing methods typically enforce formal grammars by applying global vocabulary masking at every decoding step [17], [18].”

“Outlines [23] is used to support this pattern for transformer-based LLMs.”

“To make this work, a sampling strategy is used that masks tokens which would produce invalid outputs (Willard and Louf 2023).”

“Moreover, in applications such as constrained PDFA learning and constrained or grammar-guided generation for large language models [6, 13, 14], the continuations available for comparison may be restricted by constraints imposed by the generation procedure.”

“This was followed by methods like constrained decoding [Scholak et al., 2021, Willard and Louf, 2023, Dong et al., 2025] and structured-output APIs [Qin et al., 2024, Patil et al., 2024], aiming for strict schema compliance.”

“In Section IV we address this limitation.”

“Constrained decoding methods (Koo et al., 2024; Willard & Louf, 2023; Cognetta & Okazaki, 2025) can guarantee V strict = 1 .”

“…constraints — from foundational work on incremental parsing for auto-regressive generation [ 44 ], to modern grammar-compilation approaches [ 45 ], to higher-level query languages that integrate constrained decoding into prompt programming [ 46 ] — prevent the model from emitting forbidden…”

“Outlines 52 formalised efficient guided generation as finite-state machine augmentation of the sampler, and LMQL 53 embedded constraint specification in a query language.”

“Outlines, XGrammar, and vLLM structured outputs provide constrained generation based on regular expressions, JSON schemas, context-free grammars, and related formalisms[9, 29, 30].”

“Constrained decoding [10] enforces syntactic validity but not semantic correctness.”

“The validate step itself is mechanism-agnostic: constrained decoding [15], rewrite models, or repeated sampling [16] all satisfy the contract.”

“Constrained decoding has many advantages: ( i ) modern implementations are highly efficient [6, 40], ( ii ) grammars can be reused among different LLMs, and ( iii ) it guarantees that produced samples are accepted by the grammar.”

“This was achieved using Outlines constrained generation[26].”

“Works by Willard & Louf [13] and Geng et al. [14] describe grammar-constrained decoding (GCD), where invalid tokens are excluded at inference time.”

“Sampling is not merely a post-processing detail; rather, it governs output diversity, controllability, and structural validity (Holtzman et al., 2020; Zhu et al., 2023; Willard and Louf, 2023; Dong et al., 2025).”

“The Outlines/llguidance backend used for C1+C2 [22] is fixed across all SmolLM2 cells.”

“Guidance-style systems such as Outlines [29] and later FSM-based frameworks [4, 10] make grammar and regex constraints practical for modern LLM inference.”

“10: return S This auto-generation mechanism achieves two critical objectives identified in the prompt engineering literature [5,7]: it enforces strict JSON output formatting to maximize parseability, and it provides field-level schema specification to reduce hallucination and improve extraction…”

“To help improve performance, we constrain the model to generate structured text using the Python library Outlines [7].”

“Importantly, the sub-agents have access to their associated tool (similarly to the single-agent architecture), and are able to communicate with each other through structured output [9] – the output of each LLM is constrained to follow a precise JSON-based grammar.”

“Discovering the intent vocabulary has been studied through unsupervised clustering of utterance embeddings [33, 34] and, more recently, through LLM-in-the-loop cluster evaluation and naming [12, 35], which support the iterative discovery of high-quality intents.”

“We used LLaMA-Factory [47] and incorporated Unsloth [19] for fast token streaming and Outlines [45] to impose the JSON schema as a constrained grammar, guaranteeing syntactically valid graphs even for small models.”

“SGR [1] is an architectural pattern that constrains an LLM to produce typed, schema-conformant outputs via structured output or constrained decoding [3, 14], turning domain knowledge into executable data contracts.”

“Complementary approaches include constrained decoding to enforce output structure [9] and multi-stage verifier cascades for high-stakes applications [10].”

“Insummary, our contributions are: (cid:2) A robust method for grounding symbolic representations and goals from image or textual description using VLMs. (cid:2) Extensive experimental evaluation and ablation studies demonstrating the effectiveness of the proposed approach.”

“Outlines [29] uses finite-state machines to guide to-ken sampling; LMQL [7] provides a query language for constrained outputs; SGLang [34] offers efficient structured generation with parallelism.”

“A more capable local model or constrained-decoding support [22] would likely reduce this failure rate, but at the 3B scale T6 is unreliable enough that we recommend against enabling it without output-format validation.”

“Willard et al. reformulate neural text generation as transitions between finite-state machine states (Willard and Louf 2023).”

“Then, we implement the sampling phase by integrating the LLM inference framework “Llama.cpp” under the library “llama-cpp-python” and use the library “outlines” (Willard & Louf, 2023) to perform grammar-based generation.”

“…2020); the output budget is minimal (a single JSON label), eliminating the output token dynamics that dominate inference cost for generative tasks (Willard and Louf, 2023; Zheng et al., 2024b); and SLMs can be self-hosted on commodity GPUs, avoiding API dependencies and data-residency concerns.”

“Willard et al. [13] introduced an efficient approach to guiding large language model text generation using regular expressions and context-free grammars, allowing for structured and domain-specific text generation with minimal overhead.”

“To ensure structured outputs, we adopt the outlines library [59] for constrained decoding (see Supp.”

“…decoding (GCD) (Geng et al., 2023) and its many implementations (Guidance Contributors, 2023; Park et al., 2025; Ugare et al., 2024; Willard & Louf, 2023) enforce syntactic constraints for autoregressive models by filtering out tokens that are guaranteed to eventually result in a…”

“Second, structured generation via constrained decoding forces outputs for assertion polarity, temporal anchor, and relation type into a fixed schema, eliminating the open-ended-label class of hallucinations [45,46].”

“For example, other email design platforms such as Stripo 18 , Mailchimp 19 , or Unlayer 20 employ either HTML-based editing or their own proprietary JSON models; our method could facilitate cross-platform template migration, reducing vendor lock-in and enabling interoperability between editors.”

“Constraining LLM outputs to follow specific schemas has gained attention for reliability (Willard and Louf, 2023).”

“4) Constrained Decoder Tool: We use outlines [15].”

“[19] improves LLM generation by restricting the output during decoding.”

“To enforce schema-constrained generation, we employ Outlines [27], a framework that guarantees that LLM outputs adhere to predefined formats.”

“These constraints are typically encoded via regular expressions or context-free grammars, and optimized decoding engines have emerged for deriving and applying masks (Willard & Louf, 2023; Geng et al., 2023; Park et al., 2025; Dong et al., 2024; Guidance, 2023).”

“Using this schema and prompt, we apply guided decoding [Wil23] to ensure that the LLM generates only valid region and condition indices.”

“We use constraint decoding [Willard and Louf, 2023] to limit the space of possible outputs to one of the elements in X .”

“Outlines [7] frames constrained generation as a deterministic traversal over grammar-induced automata, enabling linear-time decoding under regular or context-free grammars.”

“A common approach for these strategies biases the probabilities of the tokens in an LLM vocabulary so that restricted tokens have their probabilities set to zero when checking the previously sampled tokens with the defined grammar [7].”

“PDL types are seamlessly integrated with constrained decoding (Willard & Louf, 2023) in platforms and models that support it, to ensure the shape of the output.”

“One notable approach is Outlines, proposed in [3], which leverages finite-state machines (FSMs) to guide text generation efficiently, ensuring compliance with regular grammars while maintaining computational scalability.”

“Studies such as those by Kim Dam-yul [37] and Ayoub et al. have also evidenced [38] that it is problematic to rely on automated systems, especially when making critical decisions affecting public safety.”

“In this work, we will perform constrained generation with the help of the Outlines library Willard & Louf (2023).”

“The LLM is provided with the full alert context and execution artifacts and is required to follow strict reasoning and output constraints, including production schema immutability, conservative remediation rules, and structured JSON-only output [37], [38].”

“Structured-output and guided-generation research studies how language models can be constrained to emit outputs that satisfy regular expressions, grammar, or other structural requirements [21].”

“One option is to enforce structure externally through constrained decoding [37], [38].”