Leveraging Large Language Models to Generate Answer Set Programs
Open accessAbstract
Large language models (LLMs), such as GPT-3 and GPT-4, have demonstrated exceptional performance in various natural language processing tasks and have shown the ability to solve certain reasoning problems. However, their reasoning capabilities are limited and relatively shallow, despite the application of various prompting techniques. In contrast, formal logic is adept at handling complex reasoning, but translating natural language descriptions into formal logic is a challenging task that non-experts struggle with. This paper proposes a neuro-symbolic method that combines the strengths of large language models and answer set programming. Specifically, we employ an LLM to transform natural language descriptions of logic puzzles into answer set programs. We carefully design prompts for an LLM to convert natural language descriptions into answer set programs in a step by step manner. Surprisingly, with just a few in-context learning examples, LLMs can generate reasonably complex answer set programs. The majority of errors made are relatively simple and can be easily corrected by humans, thus enabling LLMs to effectively assist in the creation of answer set programs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A preliminary test of the approach in Ishay et al. 2023 on the benchmark dataset obtained F1-scores of 0.24 for the 8B model and 0.55 for the 70B model.
“We have performed a preliminary test of the approach in(Ishay et al. 2023) on our benchmark dataset and obtained an F1-scores of 0.24 for the 8B model and 0.55 for the 70B model.”
From Integrating Answer Set Programming and Large Language Models for Reliable Structured Knowledge Extraction (2026) · cites this paper as Ishay et al. 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Answer set programming at a glance
Brewka, G., Eiter, T., & Truszczyński, M. (2011) · Communications of the ACM · doi:10.1145/2043174.2043195 · Cited by 946 · see citing works
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · see citing works
What is answer set programming
Lifschitz, V. (2008) · National Conference on Artificial Intelligence · Cited by 330 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Selection-Inference: Exploiting Large Language Models for Interpretable Logical ReasoningPreprint
Creswell, A., Shanahan, M., & Higgins, I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.09712 · Cited by 122 · see citing works
Controlled Natural Languages for Knowledge Representation
Schwitter, R. (2010) · International Conference on Computational Linguistics · Cited by 111 · see citing works
Nye, M., Tessler, M. H., Tenenbaum, J. B., & Lake, B. M. (2021) · arXiv · doi:10.48550/arxiv.2107.02794 · Cited by 40 · see citing works
Learning to Automatically Solve Logic Grid Puzzles
Mitra, A., & Baral, C. R. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1118 · Cited by 27 · see citing works
The jobs puzzle: Taking on the challenge via controlled natural language processing
Schwitter, R. (2013) · Theory and Practice of Logic Programming · doi:10.1017/s1471068413000306 · Cited by 10 · see citing works
Baral, C. R., & Dzifcak, J. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1108.3850 · Cited by 8 · see citing works
The Jobs Puzzle: A Challenge for Logical Expressibility and Automated Reasoning
Shapiro, S. C. (2011) · Cited by 7 · see citing works
LGPSolver - Solving Logic Grid Puzzles Automatically
Jabrayilzade, E., & Tekır, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.100 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Lev, I.; MacCartney, B.; Manning, C. D.; and Levy, R. 2004. Solving logic puzzles: From robust processing to precise se- mantics. In Proceedings of the 2nd Workshop on Text Mean- ing and Interpretation, 9-16.
- Valmeekam, K.; Olmo, A.; Sreedharan, S.; and Kambham- pati, S. 2022. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.
Cited by
Sorted by citations
The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning
Cunnington, D., Law, M., Lobo, J. M., & Russo, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_5 · Cited by 16 · see citing works
Exploring formal defeasible reasoning of large language models: A Chain-of-Thought approach
Li, Z., Chen, C., Li, M., & Liao, B. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113564 · Cited by 9 · see citing works
DSPy-based neural-symbolic pipeline to enhance spatial reasoning in LLMs
Wang, R., & Sun, K. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108022 · Cited by 8 · see citing works
Visual Graph Question Answering with ASP and LLMs for Language Parsing
Bauer, J. J., Eiter, T., Higuera, N., & Oetsch, J. (2025) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.416.2 · Cited by 4 · see citing works
Using Learning from Answer Sets for Robust Question Answering with LLM
Kareem, I., Gallagher, K., Borroto, M., Ricca, F., & Russo, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-74209-5_9 · Cited by 4 · see citing works
Can Large Reasoning Models Reason about Spatial Relations?
Gardelakos, E.-O., Kyriakopoulos, V., Pantazi, D.-A., Kapopoulos, O. M., Tsourma, M., & Koubarakis, M. (2025) · ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery (GeoAI) · doi:10.1145/3764912.3770841 · Cited by 3 · see citing works
TIC: Translate-Infer-Compile for Accurate “Text to Plan” Using LLMs and Logical Representations
Agarwal, S., & Sreepathy, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_19 · Cited by 3 · see citing works
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
Le, V.-M., Tran, T. N. T., Lubos, S., Felfernig, A., & Garber, D. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779903 · Cited by 2 · see citing works
Generating Solvable and Difficult Logic Grid Puzzles
Shyne, F., Facey, K., & Cooper, S. (2024) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3638530.3654337 · Cited by 2 · see citing works
Case-Based Reasoning in Generative Agents: Review and Prospect
Zhan, H., Wu, Y., Ding, J. L., Xu, B., Yang, L., & Lin, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96559-3_12 · Cited by 1 · see citing works
Question Answering with LLMs and Learning from Answer Sets
Borroto, M., GALLAGHER, K., Ielo, A., KAREEM, I., Ricca, F., & Russo, A. (2025) · Theory and Practice of Logic Programming · doi:10.1017/s1471068425100343 · Cited by 1 · see citing works
Predicate Renaming via Large Language Models
Gentili, E., Ribeiro, T., Riguzzi, F., & Inoue, K. (2026) · Machine Learning · doi:10.1007/s10994-026-07075-1 · Cited by 1 · see citing works
Framework for Evaluating LLM Performance in Undergraduate Calculus
Dakshit, S., & Roy, S. S. (2026) · Informatics · doi:10.3390/informatics13060082 · Cited by 1 · see citing works
Alviano, M., Capalbo, M., Gottlob, G., Grillo, L., Kareem, I., Lo Scudo, F., Piccolo, S. A., & Rodriguez Reiners, L. A. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.21678
Guan, I. Y. N., Zhang, G., Liu, X., Zhao, E., & Wu, J. (2024) · International Conference on Electrical Engineering, Control and Robotics (EECR) · doi:10.1109/eecr60807.2024.10607324
Answer Set Programming and Neurosymbolic AI: Applications and Future Perspectives (Invited Talk)
Borroto, M., Ielo, A., Mazzotta, G., & Ricca, F. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-89366-7_1
Composite event recognition with arbitrary specifications
Mantenoglou, P., & Artikis, A. (2025) · Information and Computation · doi:10.1016/j.ic.2025.105373
Fine-tuning LLMs for answer set programming
Coppolillo, E., Calimeri, F., Manco, G., Perri, S., & Ricca, F. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-01017-4
ASP-Bench: From Natural Language to Logic Programs
Szeider, S. (2026) · doi:10.1145/3786168.3788402
A Diagnostic Framework for Compositional Generalization via NL-to-FOL Translation
Deveci, İ. E. (2026) · Journal of Logic Language and Information · doi:10.1007/s10849-026-09480-0
Logic-Guided Data Extraction with Answer Set Programming and Large Language Models
Alviano, M., Grillo, L., Leone, N., & Scudo, F. L. (2026) · Theory and Practice of Logic Programming · doi:10.1017/s1471068426100635
Proceedings 42nd International Conference on Logic Programming
Szeider, S. (2026) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.450.10
Distilling Answer-Set Programming Rules from LLMs for Neurosymbolic Visual Question Answering
Higuera, N., Oetsch, J., & Eiter, T. (2026) · Theory and Practice of Logic Programming · doi:10.1017/s1471068426100684
Gupta, A., Prociuk, D., Russo, A., & Delaney, B. C. (2026) · Learning Health Systems · doi:10.1002/lrh2.70114
Lightweight Language Models for Encoding ASP from Natural Language
Kareem, I., Borroto, M., Ricca, F., & Vitale, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38320-4_18
BLAST: Benchmarking LLMs with ASP-Based Structured Testing
Borroto, M., Coppolillo, E., Calimeri, F., Manco, G., Perri, S., & Ricca, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38320-4_8
- Authors
- Ishay, A., Yang, Z., & Lee, J.
- Affiliations
- Arizona State University (US)
- Samsung (South Korea) (KR)
- Venue
- Proceedings of the Twentieth International Conference on Principles of Knowledge Representation and Reasoning
- Published
- 2023-09
- Pages
- 374-383
- OpenAlex
- W4385437684
- Cited by
- 25
- References
- 14
- Citations by year
- 2026: 12 · 2025: 8 · 2024: 5
- FWCI
- 4.43
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Principles of Knowledge Representation and Reasoning (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2006747)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, answer set programming, neuro-symbolic reasoning, logic puzzles, prompt engineering, in-context learning, formal logic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
