Efficient Guided Generation for Large Language Models

“We are concerned with the problem of generating sequences of tokens from a large language model (LLM) [Vaswani et al., 2017, Radford et al., 2019] that conform to regular expressions or context-free grammars (CFGs).”

“The vocabularies, V, are composed of strings from a fixed alphabet [Sennrich et al., 2015] and N is often on the order of 104 or larger.”

“…output usable under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et…”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“To achieve this, we convert the regex into a Finite State Machine (FSM) to guide the generation process during decoding [54].”

“…note that DeAL can allow the use of a costlier model for action selection ( LLM 1 ) and a cheaper one for lookahead ( LLM 2 ). tomaton (FSA) based (Willard & Louf, 2023; Geng et al., 2023), and push-down automaton (PDA) based (Deutsch et al., 2019; Wang et al., 2023b;a), our work extends these in…”

“…biasing or masking to incorporate signals at each step, for example to ensure that the complete sequence will fall in a specified regular or context-free language (e.g., Shin et al., 2021; Scholak et al., 2021; Poesia et al., 2022; Willard & Louf, 2023; Moskal et al., 2024; Ugare et al., 2024).”

“We present three case studies of the tasks where Agent-C performs better than the baselines to describe how Agent-C outperforms other agent frameworks.”

“Some recent work as well as open source projects have proposed to use regular expressions (regex) to achieve structured generation from LLMs [23, 43].”

“…→ Character-level constraints: Enforcing character-level constraints on allowed strings is a promising area that has received much recent attention (e.g., Scholak et al., 2021; Poesia et al., 2022; Geng et al., 2023; Microsoft, 2023; Willard & Louf, 2023; Koo et al., 2024; Loula et al., 2025).”

“A similar notion of guided generation has emerged in the context of LLMs (Willard and Louf, 2023), where it typically refers to constraining the model’s output to satisfy structural requirements, such as context-free grammars, to ensure syntactic correctness for downstream applications.”

“For this reason, we used regular expression-based constrained decoding using the outlines library (Willard & Louf, 2023).”

“Engineering advances have enabled at least partial compilation of constraints expressible as membership in regular (Deutsch et al., 2019; Willard & Louf, 2023; Kuchnik et al., 2023) or context-free (Koo et al., 2024; Ugare et al., 2024; Dong et al., 2024) languages as well as restricted classes of…”

“This issue can be remedied with the Constrained Generation approach, using constrained decoding [34], [35] to alter token probabilities in a way that prevents changing the code.”

“We also observe improvements when SpotLight is used in conjunction with repeated instructions.”

“Techniques such as declarative prompting 36,37 that constraint LLM output may help avoid such scenarios.”

“Generating JSON that adheres to a specified schema is extremely important for applications like tool use and function calling (Ugare et al., 2024b; Willard & Louf, 2023).”

“Instead of free-form text generation, the model is guided to fill in required slots, adhere to the correct format, and avoid extraneous or malformed outputs [BTW23].”

“C guarantees that an autoregressive model’s generations satisfy the constraint.”

“Priority Sampling outperforms Nucleus Sampling [21] for any number of samples, reaches 91% of the autotuner improvement over -Oz in just 5 samples, and even outperforms the autotuner used to generate labels for finetuning the original model with 30 samples (Figure 3).”

“2) Tools for structured generation : Frameworks such as Outlines [47] provide tools for utilizing open-source language models to generate Cooklang-compliant recipes without the necessity for fine-tuning.”

“…for specific classes of constraints (Hokamp & Liu, 2017; Koo et al., 2024; Lu et al., 2021; 2022; Poesia et al., 2022; Post & Vilar, 2018; Ugare et al., 2024; Willard & Louf, 2023) or guiding generation with neural models (Amini et al., 2023; Kumar et al., 2022; Li et al., 2022; Qin et al., 2022).”

“We use the Outlines library (Willard & Louf, 2023) for structured generation.”

“Meanwhile, we also treat it as multilabel classification instead of 4–5 binary classification tasks to avoid repeated training for each dimension, thereby reducing training time and computational resources.”

“Reject option [13]; LLM cascades [2, 11] Cheap first stage with a native probability RL with proper scoring rewards [6, 15] Calibration is part of the product objective (RLCD) mass can depend on label wording, option order, recency, and surface form [26, 46, 51, 83, 84, 85].”

“Building on this, modules with specific functionality are invented, like unifying APIs (BerriAI, 2023), constraining LLM outputs (Willard & Louf, 2023; Liu, 2023) and parallelizing function calls (Kim et al., 2023), where APPL utilizes some of them as building blocks.”

“A complementary line of work enforces grammar or schema compliance at generation time through constrained decoding (Willard & Louf, 2023; Dong et al., 2024; Zheng et al., 2024).”

“In addition, we apply constrained generation techniques to strictly enforce this structure in the output [51].”

“On the other end are "hard" constraints which zero-out the logits of invalid tokens altogether [3].”

“Structured decoding — grammars, JSON schemas, regular-expression constraints [10, 25] — makes a generative model’s output parseable, but it remains a decoding loop, and the constraint is applied to the tokens rather than to the hypothesis space.”

“Existing frameworks differ mainly in how constraints are represented: for example, Outlines (Willard & Louf, 2023) uses regular expressions and their corresponding DFA representations, while more recent systems such as XGrammar (Dong et al., 2025) rely on context-free grammars.”

“…using In-Context Learning [3]; (2) Supervised Finetuning which enhances the correctness of the extracted outputs; (3) Structured Generation which ensures that the outputs are parse-able by downstream programs [4]; and (4) Structured Prompting which infuses layout information into the prompt [5].”

“Constrained generation has been used in code completion [9, 10, 11, 12] and structured data extraction tasks [13, 14, 15, 16, 17, 18] to incorporate a language’s structure.”

“…ones are sometimes configured, and even then greedy decoding is not fully deterministic [98] while constrained/grammar-guided generation [97] narrows toward our regime—and, non-hypothetically, durable-execution engines (Section 7.9) enforce deterministic replay in production, so within…”

“Modern structured decoding systems (Willard & Louf, 2023; Dong et al., 2025; Suresh et al., 2025; Ugare et al., 2024a;b; OpenAI, 2024) extend these ideas to JSON schemas, type systems, and domain-specific languages, achieving efficient token-level filtering through optimized finite-state machines…”

“We examined various grammar-constrained generation systems and tested them to discover their behavior during partial generation decoding (Scholak et al., 2021; Poesia et al., 2022; Willard & Louf, 2023; Ugare et al., 2025b;a; Banerjee et al., 2025; Loula et al., 2025; Suresh et al., 2025).”

“Constrained decoding is well developed as a format device [Willard and Louf, 2023], with known costs to free-form reasoning [Tam et al., 2024].”

“• Constrained Decoding : Restricting LLM output tokens to conform to grammatical or structural constraints [Willard and Louf, 2023].”

“…to drive the error particle-based guidance, which reduces error by increasing the number of particles, and chain-based guidance, which reduces TV − 1TV Hastings corrections target the same limit through mixing : the contracts geometrically in TV, yielding 𝐻 = 𝑂 (cid:0) log ( 𝛿 − 1TV )…”

“These constraints can be partially precomputed and enforced more efficiently for regular (Deutsch et al., 2019; Willard & Louf, 2023; Kuchnik et al., 2023) or context-free (Koo et al., 2024; Ugare et al., 2024b; Dong et al., 2024b; Banerjee et al., 2025; Suresh et al., 2025; Firestone et al., 2025)…”

“To understand the challenges, one can view the un-safe probability estimation as a process that independently generates multiple LLM responses for a given X , audits their safety, and computes the average of the resulting unsafe indicators.”

“Future work could leverage guided-decoding backends 37,38 that can help constrain LLM generation to a predefined JSON schema.”

“Regex-Based Decoding limited to regular expressions is supported by many frameworks (e.g., LMQL (Beurer-Kellner et al., 2023), O UTLINES (Willard & Louf, 2023), GUID - ANCE (Lundberg et al.)”

“LMQL [13] frames prompting as a query language whose output constraints (including type constraints) are enforced at decoding time, and Outlines [55] enforces output structure during generation by compiling regular expressions to finite-state machines (and grammars to pushdown automata) that mask…”

“Figure 1 gives an overview.”

“For autoregressive LLMs, a common practice is to mask out next tokens that would violate the constraint as generation proceeds left-to-right [32].”

“…decoding; type-constrained decoding [Mündler et al., 2025] reduces compilation errors by > 50% while improving functional correctness; Outlines [Willard and Louf, 2023] and XGrammar [Dong et al., 2024] together deliver up to 100 × speedup over prior constrained decoding approaches.”

“…have a long history in language modeling (Mohri, 1997) and have been adopted for constrained decoding, where an FST enforces lexical or structural constraints (Allauzen et al., 2014; Ghazvininejad et al., 2016; Stahlberg et al., 2019; Willard & Louf, 2023; Koo et al., 2024; Cognetta et al., 2025).”

“Constrained decoding modifies token-level distributions during generation, commonly to enforce syntactic formats (Willard & Louf, 2023; Beurer-Kellner et al., 2024), and has been extended to semantic and logical constraints (Nagy et al., 2026; Ma & Hu, 2025).”

“We used CFG implementations introduced by model providers or accessed via external libraries like Outlines [20] to guide the token sampling process.”

“The scientific implementation and its validation remain in the under-4 [31, 32, 40, 41], constrained decoding enforces the schema during generation when supported by the inference provider.”

“Second, LLM decoding is constrained by a finite-state machine derived from the AST JSON schema 15 , guaranteeing that every generated token sequence is a syntactically valid AST instance.”

“Vendors advertise 100% format adherence; decoder-level constrained generation enforces it mechanically [6].”

“At the decoding level, grammar-constrained generation enforces structural validity [Geng et al., 2023, Scholak et al., 2021, Willard and Louf, 2023].”

“Grammar-constrained decoding (GCD) masks tokens that would violate a target grammar, guaranteeing syntactically valid output (Geng et al., 2024; Willard & Louf, 2023; Park et al., 2025).”

“We construct a binary logit mask M ∈ {−∞ , 0 } |V| , enforcing M v = 0 if v ∈ { t pass , t fail } and −∞ otherwise [12] .”

“Inaddition Inaddition,followingtheworksonLLMs[40], weenforcetheintendedformatbyguidingthedecoderwitha statemachine.”

“Approaches such as token-level constrained decoding [83, 178, 39] and prompt-level output structuring [98] can enable LLMs to produce outputs in specific, predefined formats.”

“…[Liu et al., 2024] has led to the development of various constrained-decoding frameworks 2 , such as Guidance [Guidance AI, 2023], Outlines [Willard & Louf, 2023], XGrammar [Dong et al., 2024] and the grammar module of Llamacpp [Gerganov & al., 2023] These frameworks provide broad support…”

“…is: directly-prompted, no preceding-item context (each item in a fresh chat), constrained text generation under greedy decoding via outlines 1.2.8 [33], and standard option order, with a system message instructing the model to act as a respondent and output strictly one element of the valid…”

“A QA Planner proposes rationale-first [53], instance-level chain-of-thought [55] QA pairs targeting the dataset dimensions and tasks [52].”

“We envision token-level uncertainty as particularly impactful in the context of structured generation (Willard & Louf, 2023)—a paradigm essential for agentic workflows where reliable, syntactically correct outputs are prerequisites for effective autonomous action.”

“Modern constrained decoding engines, including Out-lines [43], XGrammar [7], and JSONFormer [33], attempt to reign in this behavior by enforcing strict syntactic schema adherence [12, 39].”

“Frameworks such as XGrammar [Dong et al., 2024] and Outlines [Willard and Louf, 2023] enforce this via local vocabulary masking : at each decoding step, the logit vector is masked to retain only tokens consistent with the grammar prefix, then renormalized.”

“Then, our framework performs (iii) constrained execution by restricting the action space at decoding time (Willard & Louf, 2023) so that only the next prescribed action on the selected path is admissible, thereby suppressing sampling errors (Section 3.3).”

“Then, for component agents, the Action Input text is parsed into structured JSON format conforming to tool-specific Pydantic schemas using the Outlines library [56], which leverages a dedicated LLM (Qwen2.”

“Grammar-constrained approaches include Outlines (Willard & Louf, 2023) and XGrammar (Dong et al., 2024).”

“Additionally, the development of structured generation paradigms [ Willard and Louf, 2023 ] has further improved the reliability of LLM-generated annotations.”

“Methods like grammar-guided constrained decoding (Willard and Louf 2023; Dong et al. 2024) can force the LLM to produce outputs that adhere to the given grammar constraints, which are therefore syntactically valid, guar-anteeing compilability.”

“Similarly, Guided Decoding frameworks (e.g., Outlines, Guidance [19]) and Schema-aware SFT methods explicitly inject grammar constraints during generation or training.”

“In the field of natural language processing, syntactic constraints have been successfully enforced by applying constrained decoding at inference time (Willard & Louf, 2023; Beurer-Kellner et al., 2023; AI, 2023).”

“We employ constrained decoding [19] to ensure comprehensive coverage through well-structured JSON outputs.”

“Guided decoding constrains a language model’s output to ensure it conforms to a specific structure, using a regular expression [36].”

“Finally, we note that other Automata-based steering methods, including industry standards like Guidance (Lundberg et al., 2024) and Outlines (Willard and Louf, 2023), are shown to be unsound through experimental results in the following section.”

“Decoding constraints provide another form of structure: TOOLDEC (Zhang et al., 2023) compiles tool syntax schemas into finite-state automata and constrains generation to syntactically valid calls, using the efficient FSM-guided generation algorithm of Outlines (Willard and Louf, 2023).”

“JSON Schema and constrained generation can keep an object well formed [2], [11], [12].”

“Constraint-decoding and grammar-guided generation methods can improve schema adherence by enforcing output structure during generation [13,14], but these meth-ods are not universally available in consumer-oriented local deployment workflows and may require separate evaluation of both structural…”

“For autoregressive generative models this distinction is crucial: the uncolored representation allows every token sequence to carry nonzero probability, elimi-1 nating the need for constrained decoding or token masking [35, 36] that would otherwise be required to enforce physicality.”

“This approach is implemented in XGrammar [18], Outlines [19], LMQL, and vLLM’s LogitsProcessor API.”

“Grammar-based constrained decoding [Willard and Louf, 2023, Ugare et al., 2024] enforces syntactic validity at generation time.”

“Outlines [20] and llguidance [21] constrain LLM output to grammars or formal languages; this guarantees structural validity but not semantic safety (a JSON-schema-valid rm -rf / is still dangerous).”

“OUTLINES , developed by Willard and Louf [2023] is one of the most commonly used tool for grammar constraind decoding generation.”

“This hybrid approach—combining statistical modeling (LLMs) with deterministic rule-based post-processing—leverages the strengths of both paradigms and improves robustness [29,30].”

“Related work has further demonstrated that regular expressions and context-free grammars can be compiled into efficient guided decoding procedures, making schema-constrained generation practical for real applications [34].”

“…works that bias LM samples towards satisfying syntactic (Ahmed et al., 2023; 2025b; Zhang et al., 2024; Geh et al., 2024; Lundberg et al., 2024; Willard & Louf, 2023; van Krieken et al., 2025; Koo et al., 2024, inter alia ), or semantic constraints (Ahmed et al., 2025a; Yidou-Weng et al.,…”

“For ML/DL, we build a deterministic transformer TT that (i) standardizes numeric features using training means/standard deviations [19], and (ii) one-hot encodes the three categorical fields with handle unknown=’ignore’.”