Efficient Guided Generation for Large Language Models

“We are concerned with the problem of generating sequences of tokens from a large language model (LLM) [Vaswani et al., 2017, Radford et al., 2019] that conform to regular expressions or context-free grammars (CFGs).”

“The vocabularies, V, are composed of strings from a fixed alphabet [Sennrich et al., 2015] and N is often on the order of 104 or larger.”

“…output usable under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et…”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“…under rigid formatting requirements that are either hard or costly to capture through fine-tuning alone [Beurer-Kellner et al., 2023, Scholak et al., 2021, Poesia et al., 2022a, Rabinovich et al., 2017, Weng, 2021, Dong et al., 2023, Poesia et al., 2022b, Geng et al., 2023, Wang et al., 2023].”

“We roughly categorize prevailing methodologies into four main groups: (i) Controlling the decoding process with external constraints, e.g. automata [24], grammar [7], and guided search [8].”

“Several methods constrain the decoding to satisfy a known grammar or JSON Schema, including Outlines [8] and Microsoft Guidance, with reported gains on structured generation tasks.”

“While O UTLINES has the lowest preprocessing offline overhead, its seconds-per-token online overhead does not meet the needs of most practical applications of LLMs.”

“A tool runtime expects a machine-readable object whose tool names, argument keys, value types, and nesting satisfy predefined interface schemas (Willard & Louf, 2023; Dong et al., 2025; Geng et al., 2025).”

“For each remaining datapoint, we compile the JSON schema into a regular expression with the Outlines library from Willard & Louf (2023) and transform that regular expression into a token-prefix automaton over the model vocabulary.”

“GCD masks illegal tokens according to a finite-state grammar [16]; it has improved structured IE in low-resource settings [17].”

“Constraining the output of the LLM to a structured JSON format suppresses variability in generation and enables programmatic validation before execution [27].”

“As a possible extension, grammar-constrained decoding could enforce the response grammar during generation [30, 31]; it would complement, rather than replace, domain validation.”

“Structured-output modes constrain decoding to a schema (Willard & Louf, 2023); prefill fixes the opening tokens of a response; and format-restriction studies document a tension between enforced form and task performance (Tam et al., 2024; Sclar et al., 2024).”

“Constrained decoding has become the standard mechanism for guaranteeing that LLM outputs conform to a schema (Willard & Louf, 2023; Geng et al., 2025).”

“…reformulate that restriction as transitions between the states of a finite-state machine, which permits an index over the vocabulary to be constructed in advance so that generation can be guided by regular expressions or context-free grammars while adding little overhead to token generation [56].”

“Deploying frontier language models often requires adapting them to narrow operational needs: shorter reasoning traces, stricter output formatting, or improved accuracy on small, confidential in-house distributions [48, 47, 21, 58, 2].”

“Structured generation [9] addresses this by constraining fixed structural elements via predefined schemas and has been shown to improve format adherence and reduce hallucination rates in clinical note generation [10].”

“Structured generation research has focused on constrained decoding and guided generation [10] to ensure syntactically valid outputs; these techniques underpin JSON mode in modern LLM APIs.”

“Several techniques compose this loop: structured generation [51], [52] forces outputs into schemas, function calling [53], [54] restricts state changes to engine-supported operations, constraint solvers [55] check levels and puzzles, simulation tests whether generated scenarios are winnable [17],…”

“Recent guided-generation work reformulates constrained neural decoding through finite-state machines and grammar constraints, improving reliability for structured interfaces [19].”

“This approach enables CFG constraints for diffusion language models and multi-region in-filling, but it requires repeated lexing, automaton construction, DFA minimization, and intersection-based emptiness checking during decoding.”

“Constrained and schema-guided generation methods [32]–[34] reduce brittleness by forcing model outputs to satisfy formal structure.”

“Outlines [8] improves the efficiency of Regex-based generation by precomputing valid token sets for each DFA state.”

“According to such typologies, tracking of public narratives related to social cohesion and democratic resilience, sits in the ‘active listening’ category, and, according to Martino seeks to enhance democratic and foreign policy objectives by promoting trust and understanding.”

“Structured constrained decoding enforces an output grammar at the token level: PICARD [23] monitors partial SQL for well-formedness; Outlines [27] formulates decoding as finite-state intersection by compiling regular expressions and context-free grammars into token-level FSMs with a vocabulary…”

“Systems such as Guidance [Microsoft Research, 2023] and Outlines [Willard and Louf, 2023] enforce constraints at the token level by masking the decoder’s output distribution.”

“Grammar-constrained decoding (GCD) [Willard and Louf, 2023, Geng et al., 2023] masks invalid tokens to guarantee structural compliance, while grammar-aligned decoding [Park et al., 2024] corrects the distribution distortion GCD introduces.”

“Grammar-constrained systems (Willard & Louf, 2023; Dong et al., 2025) and entity-constrained generation (De Cao et al., 2021) use logit masking to enforce structural constraints from a fixed grammar or candidate set. k NN-LM (Khandelwal et al., 2020) takes a softer approach, interpolating the LM’s…”

“The Instruction Generator is implemented with an MLLM backend using guided decoding [43,54] to enforce structured output formatting.”

“To transform the extracted knowledge into a queryable format, we enforce a Schema-Constraint Decoding strategy [34].”

“Frameworks for structured generation have emerged as critical infrastructure for reliable knowledge production. pydantic-ai [28] provides type-safe agents with schema validation, while constrained decoding methods [19, 41] ensure outputs conform to formal grammars.”

“This method guarantees syntactically correct, structured data and has been shown to improve task performance by eliminating parsing errors [19].”

“B. Implementation Details Constrained decoding is implemented using the Outlines 2 library [46] and classes for each setting are detailed in Sec.”

“Techniques such as few-shot prompting [Brown et al., 2020], fine-tuning [Wei et al., 2022], and prompt engineering [Schulhoff et al., 2025] steer models toward high-quality extractions, while constrained decoding and token masking can enforce structured formats [Willard and Louf, 2023].”

“Recent work on grammar-constrained generation offers promising mitigation strategies [38, 15, 29].”

“For open-weight models, we used a ReAct prompt and guided decoding to enforce a “thinking” step and JSON tool invocations (Yao et al., 2023; Willard & Louf, 2023).”

“Grammar-constrained decoding enforces well-formedness during generation itself [65], [66], [67].”

“Overall, the results suggest good reliability of the inferred posterior density, as both empirical means were included in the 94% highest density interval (HDI) of the population estimates.”

“…computation graph of generations satisfying the constraint, which we can then proceed to probabilistically reason about, exactly when possible (Ahmed et al., 2022), and otherwise approximately (Willard & Louf, 2023; Zhang et al., 2024a; Koo et al., 2024; Lundberg et al., 2024; Ahmed et al., 2025).”

“The CSC constraints are guided generation (Willard & Louf, 2023) at the level of evidence types rather than tokens.”

“…a drafter, so serving engines can treat exactness as a source of speed. stop between two values if the next step’s message has arrived 14: end for 15: poll for the next message for up to 1 s, then block number of slots, and the allocation kept is the one with the most tokens per unit of step time.”

“Constrained decoding and grammars [9, 27, 32, 34] are a tool we use, not the thesis; we measure it as one rung of the ladder (Figure 3, Table 2) and find it improves on bare prompting but not on scaffolding, and does not produce the flat curve that host resolution does.”

“The NSP selects its inputs, checks that the response has the expected type [24]–[26], and makes the call at a fixed point in the code.”

“Willard and Louf (2023) compile regular expressions into finite-state machines and Dong et al. (2025) use pushdown automata for context-free grammars; Geng et al. (2025) evaluate six frameworks over ten thousand real-world JSON Schemas and find that support for schema features varies substantially…”

“In autoregressive (AR) models, schema compliance is enforced by masking invalid next tokens for a fixed left-to-right prefix (Willard & Louf, 2023; Moskal et al., 2025; Dong et al., 2025).”

“Grammar-and schema-constrained decoding [5, 6, 11, 12] enforces validity but keeps the serial path; first-token logit selection [13] avoids decoding but aban-dons the sequential policy, inheriting label-position sensitivities [14, 15].”

“The core idea is to maintain a parser state induced by the output and, at each decoding step, mask tokens that would make the output impossible to extend into a syntactically valid sequence [26], [28].”

“Making a language model emit a well-formed structure is usually treated as a decoding problem: finite-state or grammar-constrained decoding masks the vocabulary so that only valid continuations remain [Willard and Louf, 2023, Geng et al., 2023].”

“Under such a regime the structural failures reported in Table 3 would very likely disappear for every family, as the literature on grammar-constrained generation documents for sub-7 B models [Willard and Louf, 2023; Dong et al ., 2024].”

“Grammar-constrained decoding has been shown to enforce structural validity at the token level [18– 20], while knowledge-guided approaches show that grounding LLM reasoning in structured domain knowledge improves faithfulness and reduces hallucination [21], [22].”

“This is constrained decoding , and libraries such as llguidance [23], Outlines [35] and XGrammar [11] provide it.”

“For guaranteeing schema-conformant output, constrained decoding [Willard and Louf, 2023] enforces structural validity during generation, while specialized models like NuExtract [NuMind, 2024] are purpose-built for template-guided JSON extraction.”

“Strict schema enforcement is the standard alternative; grammar-constrained decoding engines [47–49] and schema-emission benchmarks [50] map its costs, and how much output-format restrictions cost a model is actively debated [51]; for 2–9B models we adopt rescue-with-feedback instead of strict…”

“Outlines supports regular expressions and context-free grammars (Willard & Louf, 2023), SynCode provides sound and complete CFG-based masking (Ugare et al., 2025), DOMINO addresses the mismatch between grammar terminals and subword vocabularies (Beurer-Kellner et al., 2024), and XGrammar combines…”

“…the output domain but reduces accuracy, since it removes the free-text reasoning pass on which the model relies [3], even when M.-A. Kourtis and G. Xilouris are with the National Centre for Scientific Research “Demokritos” (NCSRD), Athens, Greece. the constraint itself is enforced efficiently [4].”

“…as discrete tokens [2], Kosmos-2 grounds phrases to regions [22], and Florence-2 unifies detection, captioning, and OCR in one prompt-conditioned decoder over a DaViT backbone [5,30]— as is constrained decoding for entity linking [4], text-to-SQL [24], and general grammar-constrained LLMs [28].”

“We make the latent object more specific: the intermediate is a CPDAG summary, emitted under regex-or JSON-schema-constrained decoding [Willard and Louf, 2023, Dong et al., 2024], and compared against a strong direct-answer PC baseline.”

“Grammar-constrained decoding [14] can prevent malformed syntax and, if the grammar is derived from the live registry, can prevent H1 at generation time; it does not address H4 (a syntactically perfect off-frontier call) or the H5 residue, and it couples the defense to the decoder rather than the…”

“Constraining decoding to { 0 , 1 } [13] changes nothing, so the failure at these scales is the decision, not the output format.”

“Both representations are complementary to structured-output and grammar-constrained decoding (Willard and Louf, 2023; Geng et al., 2023), which guarantee well-formed calls and can be combined with either; a constraint built from the offered set can also block unavailable functions, but it cannot…”

“Constrained decoding masks every illegal token, so a verdict is produced by construction [Willard and Louf, 2023].”

“Constrained decoding [67] makes their verdicts machine-readable, and cascades that reserve expensive models for hard inputs [ 8], or abstain when uncertain [20], trade accuracy against cost.”

“For structured output, post-hoc JSON extraction was selected instead of constrained decoding because it is model-agnostic and compatible with common local serving frameworks [22].”

“Constrained generation can improve format validity [40], but a constrained action vocabulary does not itself prove that an action was authorized.”

“To ensure the structural integrity of outputs such as JSON or code, constrained decoding frameworks like Out-lines [Willard and Louf, 2023] and XGrammar [Dong et al. , 2025] utilize grammar-based schemas to filter the model’s vocabulary at each step.”

“JSON Schema [22] provides the structural target, and constrained-decoding engines such as Outlines [23], Microsoft Guidance [24], and lm-format-enforcer [25] enforce schema conformance by masking invalid tokens during generation.”

“To prevent structural hallucinations, recent frameworks employ Logit-level constrained decoding [34], which uses a Finite State Machine to deterministically guide the VLM into generating perfectly formatted JSON.”

“The instruction-following (IF) capability of large language models (LLMs) has been studied primarily as a lever for controllability [1–3], powering applications such as formatted generation [4, 5] and tool-using agents [6–9].”

“We employ Grammar-Constrained Decoding (GCD), implemented via the Guidance framework (Willard and Louf 2023), which provides a mechanism to inject hard constraints during decoding to ensure output validity.”

“Text-level control includes prompting, retrieval [Lewis et al., 2020], structured output [Willard and Louf, 2023, Beurer-Kellner et al., 2023], and post-hoc verification [Manakul et al., 2023].”

“Efficient implementations are built on finite-state or stack-based machines, including regular-expression constraints [Willard and Louf, 2023] and high-throughput grammar engines [Dong et al., 2025].”

“Grammar-constrained decoding (GCD) [14, 15, 16], the closest to ours, masks at each decoding step any token that would violate the grammar.”

“Complementary work emphasizes auditing and verification of LLM outputs in high-stakes settings via structured outputs, validation checks, and feedback loops [24, 26], and constrained/guided decoding provides guarantees for lexical or structural constraints [17, 31].”

“Grammar Constrained Decoding (GCD) methods (Ugare et al. 2025; Dong et al. 2025; Park, Zhou, and D’Antoni 2025; Willard and Louf 2023; Beurer-Kellner, Fischer, and Vechev 2024) 1 address this by masking non-conforming log-its at each decoding step.”

“Schema-constrained and grammar-guided generation [23,7] forces LLM outputs into well-typed structures, addressing the syntactic side of structured output.”

“Traditionally, Knowledge Graph Construction (KGC) relied on pipeline-based information extraction, employing supervised models like BERT for Named Entity Recognition (NER) and Relation Extraction (RE) [ 25].”

“Structured information extraction via constrained decoding has become common practice in the past couple of years [18]– [21].”

“Scholak et al. [27], Geng et al. [25], and Willard and Louf [28] guaranteed valid output formats by removing tokens that violate a predefined structure from the probability distribution at every generation step.”

“To assess the effectiveness of the regeneration pipeline, we compared it with a general structured output method, Outlines [32].”

“Constrained generation methods such as PICARD, grammar-constrained decoding, LMQL, and guided generation reduce invalid outputs by enforcing parsers, grammars, or finite-state constraints [12]– [15].”

“Recent work has focused on grammar constrained decoding (GDC) as one specific way of enforcing syntactic constraints at the LLM decoding stage [6,22].”

“La production de sorties structurées (JSON) par un LLM peut être obtenue au décodage — génération guidée par automate ou grammaire [30, 31], qui garantit la validité syntaxique sans entraînement — ou apprise par fine-tuning, à l'instar de Gorilla [32] où un LLaMA fine-tuné dépasse GPT-4 en…”

“Constrained decoding systems such as LMQL and Outlines constrain token generation against precomputable languages [9, 10].”

“…formal grammar during generation: PICARD does so incrementally for SQL (Scholak et al., 2021), Synchromesh for several target languages via completion engines (Poesia et al., 2022), and grammar-guided generation frameworks make the hard-masking regime generic and efficient (Willard and Louf, 2023).”

“Among the patterns of § 6.2, M3 (schema-constrained generation [28]) and M4 (abstention) are the obvious next prototypes: largest reliability gain for smallest implementation cost.”