Integrating Answer Set Programming and Large Language Models for Reliable Structured Knowledge Extraction

The proposed LLMASP framework fills the void of reliable structured knowledge extraction, as demonstrated in prior work.

“Our proposed LLMASP framework fills exactly this void, as demonstrated in our prior work (Alviano, Capalbo, et al. 2026;Alviano and Grillo 2024;Alviano, Lo Scudo, et al. 2024).”

• cites Alviano, Capalbo, et al. 2026 sourceDisputed

  The Introduction (section 1; page not provided) distinguishes translating natural-language inputs into formal representations from this paper’s complementary task of making ASP solver outputs intelligible. The paper proposes a map–summarize framework for verbalizing answer sets, not an LLMASP framework for structured knowledge extraction. (LLM, conf=0.86)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites Alviano and Grillo 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Alviano, Lo Scudo, et al. 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

LLMs are better suited for data extraction than for complex reasoning, particularly in scenarios requiring grounded statements and logical inference.

“According to (Nye et al. 2021;Ren et al. 2025), LLMs are better suited for data extraction than for complex reasoning, particularly in scenarios requiring grounded statements and logical inference.”

• cites Nye et al. 2021 sourcePartly supported

  Section 3 (p. 4) uses a neural model to extract facts while a separate symbolic module checks logical constraints. The results (p. 5) report that GPT-3 paired with fact extraction and a world model outperformed GPT-3 alone on a reasoning task. This supports the distinction, but the paper does not establish that LLMs generally are better suited to data extraction than to complex reasoning across settings. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Ren et al. 2025 sourcePartly supported

  The Main Results (p. 7) report that LLM performance drops sharply on complex ASP tasks requiring multi-step logical reasoning. The paper does not compare those tasks with data extraction or establish that LLMs are better suited to extraction, and the cited passages do not address grounded statements specifically. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

A preliminary test of the approach in Ishay et al. 2023 on the benchmark dataset obtained F1-scores of 0.24 for the 8B model and 0.55 for the 70B model.

“We have performed a preliminary test of the approach in(Ishay et al. 2023) on our benchmark dataset and obtained an F1-scores of 0.24 for the 8B model and 0.55 for the 70B model.”

• cites Ishay et al. 2023 sourceSupported

The number of prompt tokens grows linearly with the number of predicates to be extracted.

“First, the number of prompt tokens grows linearly with the number of predicates to be extracted, since the pipeline issues a separate completion request for each predicate.”

LLMASP relies on multiple predicate-specific prompts, issuing a separate generation request for each predicate to be extracted.

“In particular, LLMASP relies on multiple predicate-specific prompts, issuing a separate generation request for each predicate to be extracted.”

On a maximum clique example, ChatGPT (GPT-4o and gpt-4o-mini) and Meta LLaMA 3:70b produced factually incorrect answers.

“Although the answer is fluent and grammatically well-formed, it is factually incorrect because Evan does not know Fiona.We report that we obtained a wrong answer also from Meta LLaMA 3:70b (running locally on our servers).”

JSON achieves the highest extraction quality at the expense of increased verbosity.

“Among the considered formats, CSV provides the best trade-off between accuracy and cost, JSON achieves the highest extraction quality at the expense of increased verbosity, and Datalog exhibits lower robustness in the extraction process.”

The F1-score increases from 89.2% to 96.5% with Meta LLaMA 3:70b.

“In particular, the F1-score increases from 72.1% to 86.3% when using Meta LLaMA 3:8b, and from 89.2% to 96.5% with Meta LLaMA 3:70b.”

Modeling the maximum clique problem in ASP yields a correct factual answer.

“In contrast, modeling the problem in ASP yields a correct factual answer.”

Among the considered formats, CSV provides the best trade-off between accuracy and cost.

“Among the considered formats, CSV provides the best trade-off between accuracy and cost, JSON achieves the highest extraction quality at the expense of increased verbosity, and Datalog exhibits lower robustness in the extraction process.”

Datalog exhibits lower robustness in the extraction process.

“Among the considered formats, CSV provides the best trade-off between accuracy and cost, JSON achieves the highest extraction quality at the expense of increased verbosity, and Datalog exhibits lower robustness in the extraction process.”

When generation is unconstrained, this design can incur substantial overhead in terms of completion tokens, runtime, and inference cost.

“When generation is unconstrained (i.e., when the LLM produces free-form text not guided by an explicit output grammar) this design can incur substantial overhead in terms of completion tokens, runtime, and inference cost.”

The integration of ASP consistently improves extraction accuracy over vanilla LLMs, particularly for smaller models.

“The results show that the integration of ASP consistently improves extraction accuracy over vanilla LLMs, particularly for smaller models.”

• Also stated in: § 1 Introduction · plain

  “Section 5 reports an empirical evaluation on benchmarks derived from ASP Competitions (Gebser, Maratea, et al. 2020) assessing the proposed integration of LLMs and ASP, which substantially improves fact extraction accuracy over vanilla LLM prompting.”

The F1-score increases from 72.1% to 86.3% when using Meta LLaMA 3:8b.

“In particular, the F1-score increases from 72.1% to 86.3% when using Meta LLaMA 3:8b, and from 89.2% to 96.5% with Meta LLaMA 3:70b.”

The accuracy gains from ASP integration come at a non-negligible computational and energy cost.

“Although these accuracy gains are an important contribution, they come at a non-negligible computational and energy cost.”

For A2, JSON is the only grammar that improves both macro F1 (0.740 vs. 0.721) and the perfect extraction rate (0.213 vs. 0.178) for vanilla baselines.

“For A2, JSON is the only grammar that improves both macro F1 (0.740 vs. 0.721) and the perfect extraction rate (0.213 vs. 0.178).”

Under A2, JSON attains the best overall results for LLaMA 70B.

“Under A2, JSON attains the best overall results.”

The increase in F1-score provided by the knowledge base integration is primarily driven by a noticeable reduction in false negatives.

“As shown in Appendix B, the increase in the F1-score provided by the knowledge base integration is primarily driven by a noticeable reduction in false negatives (i.e., predicates that LLMASP previously failed to extract).”

Under A1, grammar constraints lower the perfect extraction rate of LLMASP from 0.242 to 0.170 with CSV.

“Under A1, grammar constraints reduce the macro F1-score of LLMASP from 0.768 to as low as 0.699 (with Datalog) and lower the perfect extraction rate from 0.242 to 0.170 (with CSV).”

In LLM (No Info) configuration, both models exhibit a low F1-score (0.201 for 8B and 0.128 for 70B).

“In this configuration, both models exhibit a low F1-score (0.201 for 8B and 0.128 for 70B).”

For LLaMA 70B, grammar constraints (especially JSON) are consistently beneficial in tighter extraction regimes and often translate into higher perfect extraction rates.

“For LLaMA 70B, the picture reverses: grammar constraints (especially JSON) are consistently beneficial in tighter extraction regimes and often translate into higher perfect extraction rates.”

Under A2, the perfect rate for LLMASP supported by a knowledge base drops from 0.371 to 0.264.

“Under A2, the degradation in macro F1 is smaller (from 0.828 to 0.810 with JSON), yet the perfect rate still drops substantially: from 0.352 to 0.260 for LLMASP configurations without a knowledge base, and from 0.371 to 0.264 for configurations supported by a knowledge base.”

The most difficult instances for the LLMs, particularly for the 8B model, are those within the Ricochet Robots domain.

“The domain-wise analysis (Table 8 in Appendix B) reveals that the most difficult instances for the LLMs, particularly for the 8B model, are those within the Ricochet Robots domain.”

Providing the exact format of the facts to extract improves extraction performance (F1-scores of 0.686 and 0.832).

“A more substantial improvement is achieved by providing the exact format of the facts to extract, mimicking the core idea behind the A1 application file (F1-scores of 0.686 and 0.832).”

Grammar-constrained decoding consistently improves reliability by reducing malformed outputs and duplicated facts.

“The results show that grammar-constrained decoding consistently improves reliability by reducing malformed outputs and duplicated facts.”

For LLMASP with application file A1, the use of a JSON grammar increases the macro F1 from 0.912 to 0.934.

“For LLMASP with the application file A1, the use of a JSON grammar increases the macro F1 from 0.912 to 0.934 and boosts the perfect extraction rate from 0.482 to 0.580.”

On LLaMA 8B, introducing an explicit output grammar generally results in performance degradation for LLMASP variants.

“On LLaMA 8B, introducing an explicit output grammar generally results in performance degradation for LLMASP variants.”

Among the considered formats, the CSV grammar achieves the best balance between extraction quality and generation efficiency, dominating the Pareto front in most settings.

“Among the considered formats, the CSV grammar achieves the best balance between extraction quality and generation efficiency, dominating the Pareto front in most settings.”

• Also stated in: § Q5: Which grammar offers the best trade-off between extraction quality and generation efficiency? · plain

  “We empirically evaluate multiple grammar formats (JSON, Datalog, and a lightweight CSV-based representation) and show that CSV grammars offer the best trade-off between extraction accuracy and generation efficiency.”

Under A1, grammar constraints reduce the macro F1-score of LLMASP from 0.768 to as low as 0.699 with Datalog.

“Under A1, grammar constraints reduce the macro F1-score of LLMASP from 0.768 to as low as 0.699 (with Datalog) and lower the perfect extraction rate from 0.242 to 0.170 (with CSV).”

All configurations outperform the strongest baselines.

“Moving on to the extraction performance of our framework, we find that all configurations outperform the strongest baselines.”

Under A2, the perfect rate for LLMASP without a knowledge base drops from 0.352 to 0.260.

“Under A2, the degradation in macro F1 is smaller (from 0.828 to 0.810 with JSON), yet the perfect rate still drops substantially: from 0.352 to 0.260 for LLMASP configurations without a knowledge base, and from 0.371 to 0.264 for configurations supported by a knowledge base.”

For LLMASP with application file A1, the use of a JSON grammar boosts the perfect extraction rate from 0.482 to 0.580.

“For LLMASP with the application file A1, the use of a JSON grammar increases the macro F1 from 0.912 to 0.934 and boosts the perfect extraction rate from 0.482 to 0.580.”

For A1, CSV and JSON exhibit a reduction in macro F1-score for vanilla baselines (to 0.656 and 0.588, respectively).

“For A1, CSV and JSON exhibit a reduction in macro F1-score (to 0.656 and 0.588, respectively), whereas Datalog yields a slight macro-F1 improvement (0.696 vs. 0.686) but with a decrease in the perfect rate.”

LLMASP A1 achieves F1-scores of 0.768 and 0.912 for the 8B and 70B models, respectively.

“LLMASP 𝐴1 achieves F1-scores of 0.768 and 0.912 for the 8B and 70B models, respectively.”

The overall best performing behavior file for the information extraction task consists of the EIN configuration.

“We have empirically determined that the overall best performing behavior file consists of the EIN configuration; as such, we will be using the EIN behavior file in the following experiments.”

• Also stated in: § 4.3 Prompt Templates and Behavior Configurations · plain

  “In appendix A, we provide the results of our empirical investigation (and ablation study) of behavior file configurations that has led us to select the EIN configuration.”

For LLMASP with application file A1, the micro F1-score decreases slightly from 0.952 to 0.942 with JSON grammar.

“We observe that the micro F1-score decreases slightly (from 0.952 to 0.942), signaling an increase in the total number of false negatives and false positives.”

Including both descriptions and formats improves extraction performance (F1-scores of 0.721 and 0.892).

“Finally, including both descriptions and formats-mimicking the A2 application file-further improves extraction performance (F1-scores of 0.721 and 0.892).”

On configurations powered by LLaMA 8B, enforcing structured outputs is generally beneficial for suppressing hallucinations, particularly in configurations employing the A2 application file.

“On configurations powered by LLaMA 8B, enforcing structured outputs is generally beneficial for suppressing hallucinations, particularly in configurations employing the A2 application file.”

Performance under LLMASP+KB exhibits much narrower variability than in the non-KB models, especially for the 70B LLM.

“Performance under LLMASP+KB exhibits much narrower variability than in the non-KB models, especially for the 70B LLM.”

Under A2, the degradation in macro F1 for LLMASP is smaller (from 0.828 to 0.810 with JSON).

“Under A2, the degradation in macro F1 is smaller (from 0.828 to 0.810 with JSON), yet the perfect rate still drops substantially: from 0.352 to 0.260 for LLMASP configurations without a knowledge base, and from 0.371 to 0.264 for configurations supported by a knowledge base.”

CSV is the only grammar that supports a consistent reduction in energy costs and completion tokens (compared to unconstrained configurations) across all model sizes and configurations.

“Another important insight from Figure 2 is that CSV is the only grammar that supports a consistent reduction in energy costs and completion tokens (compared to the unconstrained configurations) across all model sizes and configurations.”

Domains Labyrinth, No Mystery, Ricochet Robots, and Sokoban contain instances that allow portions of the target representation to be derived via ASP once a minimal subset of facts has been extracted.

“Specifically, the domains Labyrinth (L), No Mystery (NM), Ricochet Robots (RR), and Sokoban (S) contain instances that allow portions of the target representation to be derived via ASP once a minimal subset of facts has been extracted.”

LLMASP A2 achieves F1-scores of 0.827 and 0.963.

“The application file A2 in LLMASP 𝐴2 pushes performance even further, exhibiting F1-scores of 0.827 and 0.963.”

CSV grammar consistently reduces completion tokens.

“Compact grammars, specifically CSV, consistently reduce completion tokens, directly translating to lower energy consumption and faster processing.”

Operating cost in our experiments is driven by a combination of model scale, application file configuration, and the strictness of the applied output grammar.

“Operating cost in our experiments is driven by a combination of model scale, application file configuration, and the strictness of the applied output grammar.”

In Figure 4, the Pareto front is decisively dominated by CSV-constrained setups.

“However, when isolating purely neural capabilities (Figure 4), the Pareto front is decisively dominated by CSV-constrained setups.”

Predicate-specific extraction as implemented in LLMASP consistently outperforms vanilla prompting across all evaluated models and domains.

“Across all evaluated models and domains, predicate-specific extraction as implemented in LLMASP consistently outperforms vanilla prompting.”

CSV constraints are also effective at reducing hallucinations, particularly with the larger LLaMA 70B model and richer application descriptions (A2).

“JSON-style grammar constraints yield the lowest number of hallucinated and duplicated facts across all settings, while CSV constraints are also effective, particularly with the larger LLaMA 70B model and richer application descriptions (A2).”

Energy consumption and computation time exhibit a near-perfect correlation (r=0.998, p<0.001).

“A primary observation is that energy consumption and computation time are virtually interchangeable metrics in this context; they exhibit a near-perfect correlation (𝑟 = 0.998, 𝑝 < 0.001).”

JSON is usually the most aggressive at removing duplicates and false positives.

“CSV typically maintains or boosts extraction performance while markedly cutting hallucinations, whereas JSON is usually the most aggressive at removing duplicates and false positives, occasionally resulting in a higher number of false negatives, but frequently reaching the overall best performance.”

Unconstrained LLMASP A2 (LLaMA 3 70B) achieves F1=0.963 at energy cost E=61.4×10^5 J.

“For instance, unconstrained LLMASP 𝐴2 (LLaMA 3 70B) achieves 𝐹 1 = 0.963 at an energy cost of 𝐸 = 61.4× 10 5 J, whereas [CSV] LLMASP 𝐴2 achieves comparable performance (𝐹 1 = 0.966) while reducing energy expenditure by nearly 20% (𝐸 = 49.3 × 10 5 J).”

Integrating explicit domain knowledge encoded as ASP rules with neural extraction yields further improvements in overall correctness.

“Integrating explicit domain knowledge encoded as ASP rules with neural extraction yields further improvements in overall correctness.”

Completion tokens are a dominant predictor of running time (γ2=0.0812, p<0.01).

“Crucially, Model 2 and Model 4 indicate that completion tokens (#𝑡 𝑜𝑢𝑡 ) are a dominant predictor of both energy consumption (𝛾 2 = 0.1274, 𝑝 < 0.001) and running time (𝛾 2 = 0.0812, 𝑝 < 0.01).”

JSON occasionally results in a higher number of false negatives.

“CSV typically maintains or boosts extraction performance while markedly cutting hallucinations, whereas JSON is usually the most aggressive at removing duplicates and false positives, occasionally resulting in a higher number of false negatives, but frequently reaching the overall best performance.”

The choice of output grammar radically shapes the quality-cost plane.

“Table 5 clearly demonstrates that the choice of output grammar radically shapes the quality-cost plane.”

Single-call baseline LLM A2 (LLaMA 3 70B) yields F1=0.892 at E=46.2×10^5 J.

“Even more striking is the comparison against the single-call baseline LLM 𝐴2 (LLaMA 3 70B), which yields 𝐹 1 = 0.892 at 𝐸 = 46.2× 10 5 J.”

JSON-based decoding achieves the highest extraction accuracy.

“While JSON-based decoding achieves the highest extraction accuracy, CSV emerges as the most balanced solution overall, offering strong accuracy with substantially lower energy consumption and dominating the Pareto front in most experimental settings.”

Gains in F1-score over vanilla prompting are up to 8.0 percentage points for LLaMA 3 70B.

“In particular, we observe substantial gains in F1-score over vanilla prompting, with improvements of up to 10.6 percentage points for LLaMA 3 8B and up to 8.0 percentage points for LLaMA 3 70B, confirming that structured, predicate-aware prompting is a key driver of extraction accuracy.”

The vast majority of false positives across all LLMASP and LLMASP+KB configurations are concentrated in five domains: Sokoban (S), Graph Colouring (GC), Incremental Scheduling (IS), Stable Marriage (SM), and the Valves Location Problem (VLP).

“As detailed in Appendix D, our qualitative error analysis reveals that the vast majority of false positives across all LLMASP and LLMASP+KB configurations are concentrated in five domains: Sokoban (S), Graph Colouring (GC), Incremental Scheduling (IS), Stable Marriage (SM), and the Valves Location Problem (VLP).”

JSON heavily inflates output token volume.

“Conversely, verbose formats like JSON heavily inflate output token volume, resulting in substantial energy and latency penalties despite improving structural reliability.”

With LLaMA 70B, the integration of a knowledge base has a less pronounced impact on reducing hallucinations and duplicates compared to its effect on the 8B model.

“With LLaMA 70B, the integration of a knowledge base has a less pronounced impact on reducing hallucinations and duplicates compared to its effect on the 8B model.”

This simple cleanup elevated the overall F1-score of [CSV] LLMASP A2 from 0.966 to 0.980 across the entire benchmark.

“Consequently, this simple cleanup elevated the overall F1-score of [CSV] LLMASP 𝐴2 from 0.966 to 0.980 across the entire benchmark.”

Energy and running time are almost completely determined by the number of tokens processed during inference, with model size (LLaMA 70B) acting as the primary fixed multiplier.

“The results establish that energy and running time are almost completely determined by the number of tokens processed during inference, with model size (LLaMA 70B) acting as the primary fixed multiplier.”

Gains in F1-score over vanilla prompting are up to 10.6 percentage points for LLaMA 3 8B.

“In particular, we observe substantial gains in F1-score over vanilla prompting, with improvements of up to 10.6 percentage points for LLaMA 3 8B and up to 8.0 percentage points for LLaMA 3 70B, confirming that structured, predicate-aware prompting is a key driver of extraction accuracy.”

JSON improves structural reliability.

“Conversely, verbose formats like JSON heavily inflate output token volume, resulting in substantial energy and latency penalties despite improving structural reliability.”

Integration of ASP with neural extraction leads to consistent gains in recall without degrading precision, particularly in domains where structural constraints or transitive relationships are prominent.

“Empirically, this integration leads to consistent gains in recall without degrading precision, particularly in domains where structural constraints or transitive relationships are prominent, demonstrating the effectiveness of combining neural extraction with symbolic reasoning.”

JSON frequently reaches the overall best performance.

“CSV typically maintains or boosts extraction performance while markedly cutting hallucinations, whereas JSON is usually the most aggressive at removing duplicates and false positives, occasionally resulting in a higher number of false negatives, but frequently reaching the overall best performance.”

By applying targeted rules to (i) enforce symmetry for link/2, (ii) ensure arguments of node/1 are integer IDs, and (iii) discard out-of-schema predicates and placeholder strings in IS, the total false positives for [CSV] LLMASP A2 were reduced from 2,983 to 1,053 (a 65% reduction).

“By applying targeted rules to (i) enforce symmetry for link/2, (ii) ensure arguments of node/1 are integer IDs, and (iii) discard out-of-schema predicates and placeholder strings in IS, we reduced the total false positives for [CSV] LLMASP 𝐴2 from 2,983 to 1,053 (a 65% reduction).”

JSON-style grammar constraints yield the lowest number of hallucinated and duplicated facts across all settings.

“JSON-style grammar constraints yield the lowest number of hallucinated and duplicated facts across all settings, while CSV constraints are also effective, particularly with the larger LLaMA 70B model and richer application descriptions (A2).”

The knowledge base coefficient (β3) is negative for all models.

“The coefficient for the integration of a knowledge base (𝛽 3 ) is negative for all models, suggesting that the knowledge base impacts energy consumption and execution time by reducing them, though this effect is not statistically significant.”

In Figure 3, the Pareto frontier is dominated by KB-supported configurations.

“In Figure 3, the frontier is predictably dominated by KB-supported configurations, as offloading extraction to symbolic logic structurally shrinks completion length while preserving precision.”

[CSV] LLMASP A2 exhibits a high discrepancy between D (1,986) and D_uniq (422).

“Interestingly, [CSV] LLMASP 𝐴2 exhibits a high discrepancy between D (1,986) and 𝐷 uniq (422).”

Completion tokens are a dominant predictor of energy consumption (γ2=0.1274, p<0.001).

“Crucially, Model 2 and Model 4 indicate that completion tokens (#𝑡 𝑜𝑢𝑡 ) are a dominant predictor of both energy consumption (𝛾 2 = 0.1274, 𝑝 < 0.001) and running time (𝛾 2 = 0.0812, 𝑝 < 0.01).”

Across all configurations, the inclusion of a knowledge base leads to consistent improvements in overall extraction performance.

“Across all configurations, the inclusion of a knowledge base leads to consistent improvements in overall extraction performance.”

For LLaMA 8B, KB augmentation also yields substantial gains, reaching a best F1 = 0.955 under both A1+KB and A2+KB (configuration EI).

“For LLaMA 8B, KB augmentation also yields substantial gains, reaching a best 𝐹 1 = 0.955 under both A1+KB and A2+KB (configuration EI).”

The strongest result is obtained by A1+KB in configurations such as EFN and EI, reaching (P = 0.980, R = 0.998, F1 = 0.988).

“The strongest result is obtained by A1+KB in configurations such as EFN and EI, reaching (𝑃 = 0.980, 𝑅 = 0.998, 𝐹 1 = 0.988).”

CSV emerges as the most cost-effective format, dominating the energy-performance Pareto frontier in most settings.

“Overall, CSV emerges as the most cost-effective format, dominating the energy-performance Pareto frontier in most settings.”

The improvements are consistent across predicate arities, including higher-arity facts (F1 ar3: from 0.747 to 0.902).

“the improvements are consistent across predicate arities, including higher-arity facts (𝐹 1 ar3 : from 0.747 to 0.902).”

The number of generated completion tokens serves as an excellent proxy for energy consumption.

“Because autoregressive generation scales linearly with sequence length, the number of generated completion tokens serves as an excellent proxy for energy consumption, as demonstrated by our experiments.”

• AssumptionRests on an assumption: Autoregressive generation scales linearly with sequence length.

LLMs can bypass traditional, often fragile pipelines by translating unstructured natural language directly into a rigorous set of ASP facts.

“Here, we demonstrate how LLMs bypass these traditional, often fragile pipelines by translating unstructured natural language directly into a rigorous set of ASP facts.”

The non-KB reference point ([JSON] LLMASP A2) still misses a large portion of facts (FN= 4048) despite a relatively small FP cost (FP= 488).

“The non-KB reference point ([JSON] LLMASP 𝐴2 ) still misses a large portion of facts (FN= 4048) despite a relatively small FP cost (FP= 488).”

A sensible improvement is observed for RR (Ricochet Robots), especially for the 8b model.

“We observe a sensible improvement for RR, especially for the 8b model.”

For LLaMA 70B, the non-KB A2 baseline achieves F1 ∈ [0.915, 0.972] (best at IN).

“For LLaMA 70B, the non-KB A2 baseline achieves 𝐹 1 ∈ [0.915, 0.972] (best at IN).”

A2+KB improves both precision and recall (P: from 0.871 to 0.949, R: from 0.740 to 0.963).

“A2+KB improves both precision and recall (P: from 0.871 to 0.949, R: from 0.740 to 0.963)”

Across both model sizes, a first salient effect of KB support is the near-complete elimination of redundancy: all KB configurations yield D F = 0 on these domains.

“Across both model sizes, a first salient effect of KB support is the near-complete elimination of redundancy: all KB configurations yield D 𝐹 = 0 on these domains.”

With KB support, the spread of F1 values across configurations is markedly narrower than in the non-KB A2 baseline.

“With KB support, the spread of F1 values across configurations is markedly narrower than in the non-KB A2 baseline”

By constraining the output to regular-language formats such as CSV, we achieve lightweight syntactic filtering with negligible computational overhead.

“By constraining the output to regular-language formats such as CSV (where the structure is flat, repetitive, and inherently finite-state), we achieve lightweight syntactic filtering with negligible computational overhead.”

Tested LLMs often have difficulties associating a barrier between (x-1,y) and (x,y) toward direction west with location (x-1,y).

“we observed that often the tested LLMs have difficulties to associate a barrier between (𝑥 -1, 𝑦) and (𝑥, 𝑦) toward direction west with location (𝑥 -1, 𝑦).”

Under A2+KB, CSV and Datalog are particularly economical in completion tokens and energy.

“Under A2+KB, CSV and Datalog are particularly economical (completion tokens ≈ 30k and energy ≈ 1.2-1.3× 10 5 J)”

Predicate-specific extraction combined with the selective use of explicit symbolic knowledge encoded as ASP rules is effective.

“Beyond grammar-constrained decoding, our experiments confirm the effectiveness of predicate-specific extraction combined with the selective use of explicit symbolic knowledge encoded as ASP rules.”

Grammar-constrained decoding is an effective mechanism for balancing extraction accuracy and costs within LLMASP.

“Our results show that grammar-constrained decoding is an effective mechanism for balancing extraction accuracy and costs within LLMASP.”

Under A2+KB, CSV and Datalog underperform the unconstrained KB setting in executability.

“but they underperform the unconstrained KB setting in executability (Perfect Rate is 0.055 for CSV; 0.148 for Datalog; versus 0.344 without grammar).”

Delegating the derivation of implicit knowledge to ASP improves recall without sacrificing precision.

“Delegating the derivation of implicit knowledge to ASP improves recall without sacrificing precision, highlighting the mutually reinforcing relation between neural extraction and symbolic reasoning.”

Configurations paired with A2 application files exhibit, ceteris paribus, higher performance than their A1 counterparts.

“Configurations paired with A2 application files exhibit, ceteris paribus, higher performance than their A1 counterparts.”

The single-pass LLM A2 outperforms LLMASP on extraction completeness in Sokoban.

“It is revealing that the single-pass LLM A2 outperforms LLMASP on extraction completeness in this domain.”

Summing over the false positives of S, GC, IS, and VLP accounts for 96.7% of the total false positives generated by [CSV] LLMASP A2.

“Summing over the false positives of S, GC, IS, and VLP accounts for 96.7% of the total false positives generated by [CSV] LLMASP 𝐴2 .”

A similar pattern occurs in the Valves Location Problem (VLP), where the best-performing configuration is LLM A2 (CSV).

“A similar pattern occurs in the Valves Location Problem (VLP), where the bestperforming configuration is LLM A2 (CSV).”

Across grammars, CSV and JSON exhibit the lowest performance variability across the LLMASP and LLMASP+KB configurations.

“Across grammars, CSV and JSON exhibit the lowest performance variability across the LLMASP and LLMASP+KB configurations.”

R does not help.

“while R does not”

The introduction of the multi-call LLMASP strategy leads to a decisive increase in the F1-score regardless of arity, model size, application file, and grammar.

“The introduction of the multi-call LLMASP strategy (blue for LLMASP configurations and red for LLMASP+KB) leads to a decisive increase in the F1-score regardless of arity, model size, application file, and grammar.”

Errors are highly localized.

“Notably, errors are highly localized.”

The baseline LLM with JSON and the A2 application file correctly extracts 27 out of 32 instances for PU.

“A notable exception is the baseline LLM with JSON and the A2 application file, which correctly extracts 27 out of 32 instances for PU.”

LLMASP consistently improves over the vanilla LLaMA baselines across both model sizes.

“In line with the previous experiments, Table 7 shows that LLMASP consistently improves over the vanilla LLaMA baselines across both model sizes.”

Incremental Scheduling remains the most resistant to perfect extraction.

“Incremental Scheduling remains the most resistant to perfect extraction.”

Stable Marriage (SM) reinforces this phenomenon; extraction performance degrades slightly when moving from the baseline [JSON] LLM A2 to LLMASP A2.

“Stable Marriage (SM) reinforces this phenomenon; extraction performance degrades slightly when moving from the baseline [JSON] LLM 𝐴2 to LLMASP 𝐴2 .”

The zero in the true positives for [JSON] LLM A1 in VLP is driven by the same failure mode.

“The zero in the true positives for [JSON] LLM 𝐴1 in this problem domain is driven by the same failure mode.”

The LLM A2 baseline manages to perfectly extract up to 20 out of 32 instances for Sokoban.

“Interestingly, the LLM A2 baseline manages to perfectly extract up to 20 out of 32 instances for Sokoban.”

N helps the LLMs to handle missing answers.

“N helps the LLMs to handle missing answers”

On [JSON] LLMASP A2 the F1-score increases from 0.977 to 0.985 with the post-processing rules.

“Similarly, on [JSON] LLMASP 𝐴2 the F1-score increases from 0.977 to 0.985.”

Incremental Scheduling exhibits the most chaotic error patterns, particularly with the CSV grammar, which generates out-of-schema artifacts like 1(1,one) or instances(device,quantity).

“Finally, Incremental Scheduling exhibits the most chaotic error patterns, particularly with the CSV grammar, which generates out-of-schema artifacts like 1(1,one) or instances(device,quantity).”

The failure in Graph Colouring is primarily driven by symmetry resolution errors: extracting link(82, 10) instead of link(10, 82).

“Upon contrasting the extracted predicates with the ground truth, we found this failure is primarily driven by symmetry resolution errors: extracting link(82, 10) instead of link(10, 82).”

Under A1, the 70B model rises from F1 = 0.904 to F1 = 0.968 with EIN.

“while the 70B model rises from 𝐹 1 = 0.904 to 𝐹 1 = 0.968 with EIN”

For the larger model, baseline quality is already near saturation on the four selected domains (LLMASP A2 with JSON: F1 macro = 0.983, Perfect Rate = 0.719).

“For the larger model, baseline quality is already near saturation on the four selected domains (LLMASP 𝐴2 with JSON: 𝐹 1 macro = 0.983, Perfect Rate= 0.719; Table 10).”

The [CSV] LLMASP A1 configuration struggles slightly more with unary predicates, occasionally outputting string variables like node(node_12) or node(node27) instead of the required integer ID.

“The [CSV] LLMASP 𝐴1 configuration struggles slightly more with unary predicates, occasionally outputting string variables like node(node_12) or node(node27) instead of the required integer ID.”

LLMASP reverses the pattern observed in single-call baselines: in most configurations, the highest F1-score is obtained for unary predicates, followed by binary and ternary predicates.

“Additionally, LLMASP reverses the pattern observed in single-call baselines: in most configurations, the highest F1-score is obtained for unary predicates, followed by binary and ternary predicates.”

This is a different error mode than the one observed, on the same problem domain, for [CSV] LLMASP A2, where the primary error mode consists of the symmetry resolution of the link(node_1,node_2) predicate.

“Additionally, this is a different error mode than the one observed, on the same problem domain, for [CSV] LLMASP 𝐴2 , where the primary error mode consists of the symmetry resolution of the link(node_1,node_2) predicate.”

For Sokoban, the primary source of error is the hallucination of the stone/1 predicate.

“For Sokoban, the primary source of error is the hallucination of the stone/1 predicate.”

Under A2, EIN on the 70B model attains the highest aggregate performance (P = 0.974, R = 0.988, F1 = 0.981).

“Under A2, the same trend persists and culminates in the overall best configuration, namely EIN on the 70B model, which attains the highest aggregate performance (𝑃 = 0.974, 𝑅 = 0.988, 𝐹 1 = 0.981).”

Under A1, the 8B model improves from the best vanilla setting (Descr.+Format, F1 = 0.737) to F1 = 0.823 with EIN.

“Under A1, the 8B model improves from the best vanilla setting (Descr.+Format,𝐹 1 = 0.737) to 𝐹 1 = 0.823 with EIN”

The JSON configuration introduces a unique failure mode: it hallucinates the arity integer directly into the predicate name (e.g., pipe/2(101,103) and junction/1(...).

“The JSON configuration introduces a unique failure mode here: it hallucinates the arity integer directly into the predicate name (e.g., pipe/2(101,103) and junction/1(”

E tends to improve the F1-score.

“E tends to improve the F1-score.”

Single-call baselines (LLM) exhibit a lower F1-score for unary predicates (arity 1), independent of model size and application files.

“The first notable trend is that single-call baselines (LLM, shown in green in Figure 8) exhibit a lower F1-score for unary predicates (arity 1), independent of model size and application files.”

LLMASP A2 configurations (70B) already successfully extract over 23,500 true positives out of the 23,760 total facts in the database.

“As shown in Table 4, LLMASP A2 configurations (70B) already successfully extract over 23,500 true positives out of the 23,760 total facts in our database.”

In the setting of unconstrained generation, even correct facts become costly to recover reliably.

“In this setting, even correct facts become costly to recover reliably.”

Combining ASP with grammar constrained LLM output yields a reliable, scalable, and cost-effective approach to structured knowledge extraction.

“Overall, the results demonstrate that combining ASP with grammar constrained LLM output yields a reliable, scalable, and cost-effective approach to structured knowledge extraction.”

These results demonstrate the effectiveness of ASP-guided extraction in improving correctness, especially for larger models.

“These results demonstrate the effectiveness of ASP-guided extraction in improving correctness, especially for larger models.”

ASP excels on logical and relational reasoning tasks that remain challenging even for the most advanced LLMs.

“That is, ASP excels precisely on those logical and relational reasoning tasks that remain challenging even for the most advanced LLMs (Li et al. 2024), highlighting the complementarity between the two paradigms.”

• cites Li et al. 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

By interleaving answer-like natural language with inferred facts, the model departs from the intended task of request representation and introduces spurious domain knowledge.

“By interleaving answer-like natural language with inferred facts, the model departs from the intended task of request representation and introduces spurious domain knowledge.”

ASP and LLMs offer complementary strengths in symbolic reasoning and natural language understanding.

“Answer Set Programming (ASP) and Large Language Models (LLMs) offer complementary strengths in symbolic reasoning and natural language understanding.”

• Also stated in: § 1 Introduction · plain

  “Large Language Models (LLMs) and Answer Set Programming (ASP) represent two distinct but complementary paradigms in Artificial Intelligence (AI).”

• cites Li et al. 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The inclusion of a knowledge base might render extensive prompt engineering techniques less relevant when deploying large models within the LLMASP framework.

“This suggests that the inclusion of a knowledge base might render extensive prompt engineering techniques less relevant when deploying large models within the LLMASP framework.”

The generation of verbose or nested JSON structures increases the number of completion tokens, leading to higher computational cost without a proportional gain in expressiveness for this task.

“The generation of verbose or nested JSON structures increases the number of completion tokens, leading to higher computational cost without a proportional gain in expressiveness for this task.”

During generation, the decoder consults the grammar at each step to determine the set of admissible next tokens, pruning all others from the search space.

“During generation, the decoder consults the grammar at each step to determine the set of admissible next tokens, pruning all others from the search space.”

• cites Geng et al. 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The difference in token counts illustrates the higher syntactic overhead introduced by JSON for simple relational data.

“This difference illustrates the higher syntactic overhead introduced by JSON for simple relational data.”

Tighter admissible output spaces better support predicate-specific, slot-filling behavior with limited generative effort.

“This suggests that tighter admissible output spaces better support predicate-specific, slot-filling behavior with limited generative effort.”

For the 70B model, the JSON grammar provides the most robust improvements across all configurations.

“For the 70B model, the JSON grammar provides the most robust improvements across all configurations.”

Not all structured formats uniformly constrain the generator under weaker prompting.

“With A1, the gains are smaller but still visible: JSON and Datalog reduce FP relative to LLM 𝐴1 , whereas CSV increases FP, indicating that not all structured formats uniformly constrain the generator under weaker prompting.”

Grammar-constrained generation does not pay off for the 8B model.

“In general, grammar-constrained generation does not pay off for the 8B model.”

The effectiveness of grammar-constrained generation depends jointly on model capacity, the choice of the grammar, and the application file adopted.

“Overall, the results indicate that the effectiveness of grammar-constrained generation depends jointly on model capacity, the choice of the grammar, and the application file adopted.”

On smaller models, the benefits of grammar-constrained decoding are neither systematic nor guaranteed, but instead depend strongly on the interaction between grammar choice, configuration, and extraction strategy.

“Overall, these results indicate that, on smaller models, the benefits of grammar-constrained decoding are neither systematic nor guaranteed, but instead depend strongly on the interaction between grammar choice, configuration, and extraction strategy.”

Integrating grammar-constrained decoding into LLMASP substantially improves the reliability and scalability of structured knowledge extraction.

“Overall, these results demonstrate that integrating grammar-constrained decoding into LLMASP substantially improves the reliability and scalability of structured knowledge extraction.”

• AssumptionRests on an assumption: The benchmarks derived from ASP Competitions are representative of structured knowledge extraction tasks in general.

When the generator is noisy, downstream reasoning can amplify the volume of candidate facts and their repetitions.

“Without an output grammar, LLMASP 𝐴1 on 8B exhibits substantial redundancy (D = 4,556) alongside elevated FP (7,103), suggesting that when the generator is noisy, downstream reasoning can amplify the volume of candidate facts and their repetitions.”

While JSON is expressive and widely adopted, it introduces substantial syntactic overhead for simple relational data such as ASP facts, which are typically flat and homogeneous.

“While JSON is expressive and widely adopted, it introduces substantial syntactic overhead for simple relational data such as ASP facts, which are typically flat and homogeneous.”

Grammar-constrained decoding is particularly well suited for structured output tasks such as information extraction and symbolic representation, where correctness depends not only on semantic content but also on adherence to a precise format.

“Grammar-constrained decoding is particularly well suited for structured output tasks such as information extraction and symbolic representation, where correctness depends not only on semantic content but also on adherence to a precise format.”

Employing the CSV grammar is the most cost-effective approach.

“Combining these results with our previous analysis of extraction performance yields a critical insight: employing the CSV grammar is the most cost-effective approach.”

An increase of 100,000 completion tokens causes an 8.5% increase in running time.

“Putting these numbers into perspective, given that token counts are scaled by 10 5 , an increase of 100, 000 completion tokens causes a 13.6% increase in energy consumption and an 8.5% increase in running time.”

[CSV] LLMASP A2 delivers state-of-the-art extraction performance for an energy increase of just 6.7% over the single-call baseline.

“In effect, [CSV] LLMASP 𝐴2 delivers state-of-the-art extraction performance for an energy increase of just 6.7% over the single-call baseline.”

Structured, predicate-aware prompting is a key driver of extraction accuracy.

“In particular, we observe substantial gains in F1-score over vanilla prompting, with improvements of up to 10.6 percentage points for LLaMA 3 8B and up to 8.0 percentage points for LLaMA 3 70B, confirming that structured, predicate-aware prompting is a key driver of extraction accuracy.”

Optimizing for energy efficiency simultaneously yields equivalent reductions in runtime latency.

“Therefore, optimizing for energy efficiency simultaneously yields equivalent reductions in runtime latency.”

There are certain instances in the dataset where the LLM struggles particularly under the CSV grammar.

“This suggests that there are certain instances in our dataset where the LLM struggles particularly under the CSV grammar.”

An increase of 100,000 completion tokens causes a 13.6% increase in energy consumption.

“Putting these numbers into perspective, given that token counts are scaled by 10 5 , an increase of 100, 000 completion tokens causes a 13.6% increase in energy consumption and an 8.5% increase in running time.”

A reduction of 100,000 completion tokens yields a 7.8% faster execution time.

“Conversely, focusing on reducing the number of completion tokens as a mitigation strategy, a reduction of 100, 000 completion tokens yields a 12.0% reduction in energy consumption and a 7.8% faster execution time.”

JSON's inflation of output token volume results in substantial energy and latency penalties.

“Conversely, verbose formats like JSON heavily inflate output token volume, resulting in substantial energy and latency penalties despite improving structural reliability.”

Energy consumption and computation time are virtually interchangeable metrics in this context.

“A primary observation is that energy consumption and computation time are virtually interchangeable metrics in this context; they exhibit a near-perfect correlation (𝑟 = 0.998, 𝑝 < 0.001).”

Focusing on reducing the number of output tokens is a highly effective strategy for curbing resource demands.

“This reveals that focusing on reducing the number of output tokens is a highly effective strategy for curbing resource demands.”

CSV and JSON lead to distinct error patterns.

“CSV and JSON lead to distinct error patterns: CSV typically maintains or boosts extraction performance while markedly cutting hallucinations, whereas JSON is usually the most aggressive at removing duplicates and false positives, occasionally resulting in a higher number of false negatives, but frequently reaching the overall best performance.”

As a symbolic tool for the exact generation of facts, the knowledge base inherently helps to suppress hallucinations and duplicates.

“Indeed, as a symbolic tool for the exact generation of facts, the knowledge base inherently helps to suppress hallucinations and duplicates.”

Strong models benefit from grammar-constrained generation by reducing redundancy and sharpening extraction accuracy.

“Taken together, these findings suggest that (i) strong models benefit from grammar-constrained generation in that it reduces redundancy and sharpens extraction accuracy;”

Grammar-constrained decoding is an effective and principled mechanism for controlling both the accuracy and the computational footprint of LLMASP.

“All in all, these results demonstrate that grammar-constrained decoding is an effective and principled mechanism for controlling both the accuracy and the computational footprint of LLMASP.”

A reduction of 100,000 completion tokens yields a 12.0% reduction in energy consumption.

“Conversely, focusing on reducing the number of completion tokens as a mitigation strategy, a reduction of 100, 000 completion tokens yields a 12.0% reduction in energy consumption and a 7.8% faster execution time.”

These errors are superficial extraction artifacts rather than systemic reasoning failures.

“Crucially, these errors are superficial extraction artifacts rather than systemic reasoning failures.”

These failure modes can be easily mitigated through lightweight post-processing.

“We demonstrate that these failure modes can be easily mitigated through lightweight post-processing.”

Combining neural extraction with symbolic reasoning is effective.

“Empirically, this integration leads to consistent gains in recall without degrading precision, particularly in domains where structural constraints or transitive relationships are prominent, demonstrating the effectiveness of combining neural extraction with symbolic reasoning.”

For LLaMA 8B, KB support fundamentally changes the error profile from omission-dominated to near-complete coverage.

“For LLaMA 8B, KB support fundamentally changes the error profile from omission-dominated to near-complete coverage.”

Once the LLM extracts a compact set of parameters (e.g., grid size, step bounds, or structural constraints), ASP reconstructs the expanded relational structure deterministically, with limited semantic ambiguity.

“once the LLM extracts a compact set of parameters (e.g., grid size, step bounds, or structural constraints), ASP reconstructs the expanded relational structure deterministically, with limited semantic ambiguity.”

The additional domain description in A2 remains valuable when the underlying LLM has limited capacity.

“suggesting that the additional domain description in A2 remains valuable when the underlying LLM has limited capacity.”

Stricter serialization can make seed extraction more brittle for a low-capacity model.

“consistent with the interpretation that stricter serialization can make seed extraction more brittle for a low-capacity model.”

The A2 description remains important to ensure accurate extraction of the seed facts on which KB closure critically depends.

“This highlights a key constraint for low-capacity models: the A2 description remains important to ensure accurate extraction of the seed facts on which KB closure critically depends.”

This successfully reconciles the traditionally opposing objectives of reducing energy expenditure and maintaining high performance.

“This successfully reconciles the traditionally opposing objectives of reducing energy expenditure and maintaining high performance, making it highly effective for scalable symbolic pipelines like LLMASP.”

Neural extraction and symbolic reasoning are mutually reinforcing.

“Delegating the derivation of implicit knowledge to ASP improves recall without sacrificing precision, highlighting the mutually reinforcing relation between neural extraction and symbolic reasoning.”

Our grammars strictly define regular languages devoid of recursion or deep nesting.

“Although our constraints are expressed using standard BNF notation, the grammars strictly define regular languages devoid of recursion or deep nesting.”

LLMASP is a modular neuro-symbolic framework that integrates Large Language Models with Answer Set Programming to support reliable extraction of structured facts from natural language and their subsequent use for logical reasoning.

“In this article, we presented LLMASP, a modular neuro-symbolic framework that integrates Large Language Models with Answer Set Programming to support reliable extraction of structured facts from natural language and their subsequent use for logical reasoning.”

Operational cost can decrease because the model is tasked with producing only a compact set of explicit seed facts, while the ASP component deterministically reconstructs the implied closure.

“Second, operational cost can decrease because the model is tasked with producing only a compact set of explicit seed facts, while the ASP component deterministically reconstructs the implied closure.”

When missing facts are ASP-derivable, quality improves by reducing omission errors without requiring the LLM to explicitly enumerate all consequences.

“First, when missing facts are ASP-derivable, quality improves by reducing omission errors without requiring the LLM to explicitly enumerate all consequences.”

Minimizing completion tokens offers a direct, operational strategy for cost reduction.

“Consequently, minimizing completion tokens offers a direct, operational strategy for cost reduction.”

The extracted representation effectively acts as a factual database over which custom, pre-defined ASP programs can operate.

“This extracted representation effectively acts as a factual database over which custom, pre-defined ASP programs can operate.”

ASP-based inference primarily recovers facts that are logically entailed by the extracted information but are not explicitly mentioned in the input text, rather than introducing spurious or hallucinated facts.

“This indicates that ASP-based inference primarily recovers facts that are logically entailed by the extracted information but are not explicitly mentioned in the input text, rather than introducing spurious or hallucinated facts.”

We enriched the LLMASP architecture by introducing a grammar-constrained decoding module that enforces syntactic validity of generated outputs through formal grammars.

“We enriched the LLMASP architecture by introducing a grammar-constrained decoding module that enforces syntactic validity of generated outputs through formal grammars.”

Stronger structural constraints can guide large models toward more faithful and stable completions.

“JSON-based configurations, although more verbose and energy consuming, consistently yield among the highest accuracy and the lowest hallucination rates, suggesting that stronger structural constraints can guide large models toward more faithful and stable completions.”

For the smaller model, KB support induces a substantial upward shift in extraction quality.

“For the smaller model, KB support induces a substantial upward shift in extraction quality.”

Our constraints can be enforced using finite-state automata (FSA).

“As such, our constraints can be enforced using finite-state automata (FSA).”

With a few exceptions, the use of ASP is beneficial for the generation process.

“With a few exceptions, the use of ASP is beneficial for the generation process.”

Our approach structurally enforces efficiency at generation time.

“Unlike these methods, which attempt to compress outputs post-generation or rely on heuristic token budgets, our approach structurally enforces efficiency at generation time.”

The KB predominantly addresses omission errors by completing derivable structure.

“reinforcing the interpretation that the KB predominantly addresses omission errors by completing derivable structure.”

The benefit of predicate-focused prompting extends to both capacity regimes.

“indicating that the benefit of predicate-focused prompting extends to both capacity regimes.”

Even when F1 cannot improve materially because performance is already close to the ceiling, ASP-based completion remains a strong computational lever by avoiding generative enumeration of implicit consequences.

“Thus, even when F1 cannot improve materially because performance is already close to the ceiling, ASP-based completion remains a strong computational lever by avoiding generative enumeration of implicit consequences.”

Forcing synthesis models into an extraction role yields malformed programs or invalid predicate formats.

“Forcing synthesis models into an extraction role yields malformed programs or invalid predicate formats.”

As in the non-KB setting, grammars induce differentiated operating points.

“As in the non-KB setting, grammars induce differentiated operating points.”

Overall, the EIN behavior file yields the most effective and stable performance within LLMASP.

“Overall, these results indicate that the EIN behavior file yields the most effective and stable performance within LLMASP”

The multi-call strategy of LLMASP inadvertently forces the model to over-analyze, deriving implicit entailments rather than sticking to strict textual extraction.

“Here, the multi-call strategy of LLMASP inadvertently forces the model to over-analyze, deriving implicit entailments rather than sticking to strict textual extraction.”