CoChat
Sign Up
PreprintCited by 1Logic, Reasoning, and KnowledgeAI-based Problem Solving and PlanningNatural Language Processing Techniques

Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)

Open access
Ren, Lin, Xiao, Guohui, Qi, Guilin, Geng, Yishuai, Xue, Haohan·2025·arXiv
DOI 10.48550/arxiv.2507.19749arXiv 2507.19749
Open this paper in CoChat — free accountOpen PDF

Abstract

Answer Set Programming (ASP) is a powerful paradigm for non-monotonic reasoning. Recently, large language models (LLMs) have demonstrated promising capabilities in logical reasoning. Despite this potential, current evaluations of LLM capabilities in ASP are often limited. Existing works normally employ overly simplified ASP programs, do not support negation, disjunction, or multiple answer sets. Furthermore, there is a lack of benchmarks that introduce tasks specifically designed for ASP solving. To bridge this gap, we introduce ASPBench, a comprehensive ASP benchmark, including three ASP specific tasks: ASP entailment, answer set verification, and answer set computation. Our extensive evaluations on ASPBench reveal that while 14 state-of-the-art LLMs, including \emph{deepseek-r1}, \emph{o4-mini}, and \emph{gemini-2.5-flash-thinking}, perform relatively well on the first two simpler tasks, they struggle with answer set computation, which is the core of ASP solving. These findings offer insights into the current limitations of LLMs in ASP solving. This highlights the need for new approaches that integrate symbolic reasoning capabilities more effectively. The code and dataset are available at https://github.com/HomuraT/ASPBench.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.19749

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs are better suited for data extraction than for complex reasoning, particularly in scenarios requiring grounded statements and logical inference.

    “According to (Nye et al. 2021;Ren et al. 2025), LLMs are better suited for data extraction than for complex reasoning, particularly in scenarios requiring grounded statements and logical inference.”

    From Integrating Answer Set Programming and Large Language Models for Reliable Structured Knowledge Extraction (2026) · cites this paper as Ren et al. 2025

    The Main Results (p. 7) report that LLM performance drops sharply on complex ASP tasks requiring multi-step logical reasoning. The paper does not compare those tasks with data extraction or establish that LLMs are better suited to extraction, and the cited passages do not address grounded statements specifically. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works

    “Test-time scaling (Muennighoff et al. 2025) and model scaling (Kaplan et al. 2020) are two approaches to improve the performance of LLMs.”

Flagged first, then by citations

Research works printed but not linked: 50

  1. Allaway, E.; Hwang, J. D.; Bhagavatula, C.; et al. 2023. Penguins don't fly: Reasoning about generics through instantiations and exceptions. In Proc. of EACL'2023 , 26102627.
  2. Alviano, M.; Calimeri, F.; Dodaro, C.; et al. 2017. The ASP system DLV2. In Proc. of LPNMR'2017 , 215-221.
  3. Apt, K. R.; Blair, H. A.; and Walker, A. 1988. Towards a theory of declarative knowledge. In Foundations of deductive databases and logic programming . Elsevier. 89-148.
  4. Barrett, C.; Stump, A.; and Tinelli, C. 2010. The smtlib standard: Version 2.0. In Proceedings of the 8th international workshop on satisfiability modulo theories (Edinburgh, UK) , volume 13, 14.
  5. Ben-Eliyahu, R., and Dechter, R. 1994. Propositional semantics for disjunctive logic programs. Annals of Mathematics and Artificial intelligence 12:53-87.
  6. Borroto, M.; Kareem, I.; and Ricca, F. 2024. Towards automatic composition of asp programs from natural language specifications. arXiv preprint arXiv:2403.04541 . · arXiv
  7. Chan, J.; Gaizauskas, R. J.; and Zhao, Z. 2025. RULEBREAKERS: Challenging llms at the crossroads between formal logic and human-like reasoning. In Proc. of ICML'2025 .
  8. Chen, M.; Li, G.; Wu, L. I.; et al. 2024a. Can language models pretend solvers? logic code simulation with llms. In Proc. of SETTA'2024 , 102-121.
  9. Chen, M.; Li, G.; Wu, L. I.; et al. 2024b. Can language models pretend solvers? logic code simulation with LLMs. arXiv preprint arXiv:2403.16097 . · arXiv
  10. Clark, P.; Tafjord, O.; and Richardson, K. 2021. Transformers as soft reasoners over language. In Proc. of the IJCAI'2021 , 3882-3890.
  11. Darwiche, A., and Pearl, J. 1997. On the logic of iterated belief revision. Artif. Intell. 89(1-2):1-29.
  12. Feng, J.; Xu, R.; Hao, J.; et al. 2023. Language models can be logical solvers. arXiv preprint arXiv:2311.06158 . · arXiv
  13. Forgy, C. L. 1989. Rete: A fast algorithm for the many pattern/many object pattern match problem. In Readings in artificial intelligence and databases . Elsevier. 547-559.
  14. Fu, Y.; Peng, H.; Ou, L.; et al. 2023. Specializing smaller language models towards multi-step reasoning. In Proc. of ICML'2023 , 10421-10430.
  15. Gelfond, M., and Lifschitz, V. 1988. The stable model semantics for logic programming. In Proc. of ICLP/SLP'1988 , 1070-1080.
  16. Guo, D.; Yang, D.; Zhang, H.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 . · arXiv
  17. Huang, J., and Chang, K. C.-C. 2023. Towards reasoning in large language models: A survey. In Proc. of Findings of ACL'2023 , 1049-1065.
  18. Ishay, A.; Yang, Z.; and Lee, J. 2023. Leveraging large language models to generate answer set programs. arXiv preprint arXiv:2307.07699 . · arXiv
  19. Jiang, J.; Wang, F.; Shen, J.; et al. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 . · arXiv
  20. Josephson, J. R., and Josephson, S. G. 1996. Abductive inference: Computation, philosophy, technology . Cambridge University Press.
  21. Leidinger, A.; Rooij, R. V.; and Shutova, E. 2024. Are LLMs classical or nonmonotonic reasoners? lessons from generics. In Proc. of ACL'2024 , 558-573.
  22. Li, Z.; Cao, Y.; Xu, X.; Jiang, J.; et al. 2024. LLMs for relational reasoning: How far are we? In Proc. of LLM4CODE@ICSE'2024 , 119-126.
  23. Liu, A.; Feng, B.; Xue, B.; et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 . · arXiv
  24. Liu, H.; Fu, Z.; Ding, M.; et al. 2025. Logical reasoning in large language models: A survey. arXiv preprint arXiv:2502.09100 . · arXiv
  25. Lyu, C.; Yan, L.; Xing, R.; et al. 2024. Large language models as code executors: An exploratory study. arXiv preprint arXiv:2410.06667 . · arXiv
  26. Moura, L. D., and Bjørner, N. 2008. Z3: An efficient SMT solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems , 337340. Springer.
  27. Muennighoff, N.; Yang, Z.; Shi, W.; et al. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 . · arXiv
  28. Niemel¨ a, I. 1999. Logic programs with stable model semantics as a constraint programming paradigm. Annals of mathematics and Artificial Intelligence 25:241-273.
  29. Parmar, M.; Patel, N.; Varshney, N.; et al. 2024. LogicBench: Towards systematic evaluation of logical reasoning ability of large language models. In Proc. of ACL'2024 , 13679-13707.
  30. Perak, B.; Beliga, S.; and Meˇ strovi´ c, A. 2024. Into LLM using RAG incorporating dialect understanding and prompt engineering techniques for causal commonsense reasoning. In Eleventh Workshop on NLP for Similar Languages, Varieties, Proc. of the and Dialects (VarDial'2024) , 220-229.
  31. Reiter, R. 1980. A logic for default reasoning. Artif. Intell. 13(1-2):81-132.
  32. Reiter, R. 1988. Nonmonotonic reasoning. In Exploring artificial intelligence . Elsevier. 439-481.
  33. Rudinger, R.; Shwartz, V.; Hwang, J. D.; et al. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Proc. of Findings of ACL'2020 , 4661-4675.
  34. Saha, S.; Yu, X. V.; Bansal, M.; et al. 2023. MURMUR: Modular multi-step reasoning for semi-structured data-totext generation. In Proc. of Findings of ACL'2023 , 1106911090.
  35. Srivatsa, K. A., and Kochmar, E. 2024. What makes math word problems challenging for LLMs? In Proc. of Findings of NAACL'2024 , 1138-1148.
  36. Tafjord, O.; Dalvi, B.; and Clark, P. 2021. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Proc. Findings of ACL'2021 , volume ACL/IJCNLP 2021 of Findings of ACL , 3621-3634. Association for Computational Linguistics.
  37. Tam, Z. R.; Wu, C.-K.; Tsai, Y.-L.; et al. 2024. Let me speak freely? a study on the impact of format restrictions on performance of large language models. arXiv preprint arXiv:2408.02442 . · arXiv
  38. Tian, Y.; Zhang, F.; and Peng, N. 2023. Harnessing blackbox control to boost commonsense in LM's generation. In Proc. of EMNLP'2023 , 5417-5432.
  39. Wang, K.; Qi, G.; Li, J.; et al. 2024a. Can large language models understand dl-lite ontologies? an empirical study. In Proc. of Findings of EMNLP'2024 , 2503-2519.
  40. Wang, K.; Qi, G.; Li, J.; et al. 2024b. Can large language models understand DL-Lite ontologies? an empirical study. Proc. of Findings of EMNLP'2024 .
  41. Wang, S.; Wei, Z.; Choi, Y.; et al. 2024c. Can llms reason with rules? logic scaffolding for stress-testing and improving llms. In Proc. of ACL'2024 , 7523-7543.
  42. Wang, S.; Wei, Z.; Choi, Y.; et al. 2024d. Can LLMs reason with rules? logic scaffolding for stress-testing and improving LLMs. In Proc. of ACL'2024 , 7523-7543.
  43. Wang, W.; Liu, K.; Chen, A. R.; et al. 2024e. Python symbolic execution with llm-powered code generation. arXiv preprint arXiv:2409.09271 . · arXiv
  44. Xiu, Y.; Xiao, Z.; and Liu, Y. 2022. LogicNMR: Probing the non-monotonic reasoning ability of pre-trained language models. In Proc. of Findings of EMNLP'2022 , 3616-3626.
  45. Yang, K., et al. 2024. If LLM is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. arXiv preprint arXiv:2401.00812 . · arXiv
  46. Yang, A.; Li, A.; Yang, B.; et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388 . · arXiv
  47. Yang, A.; Yu, B.; Li, C.; Liu, D.; et al. 2025b. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383 . · arXiv
  48. Yue, M.; Zhang, Y.; Liu, J.; et al. 2025. A survey of large language model agents for question answering. arXiv preprint arXiv:2503.19213 . · arXiv
  49. Zheng, L. M.; Chiang, W.-L.; Sheng, Y.; et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Proc. of NeurIPS'2023 46595-46623.
  50. Zhu, Y.; Yuan, H.; Wang, S.; et al. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107 . · arXiv

Cited by

Sorted by citations

Authors
Ren, L., Xiao, G., Qi, G., Geng, Y., & Xue, H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.19749
arXiv
2507.19749
OpenAlex
W6891825114
Cited by
0
References
1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
answer set programming, large language models, benchmarking, answer set computation, nonmonotonic reasoning, symbolic reasoning, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review