Inductive Learning of Logical Theories with LLMs: A Expressivity-graded Analysis
Open accessAbstract
This work presents a novel systematic methodology to analyse the capabilities and limitations of Large Language Models (LLMs) with feedback from a formal inference engine, on logic theory induction. The analysis is complexity-graded w.r.t. rule dependency structure, allowing quantification of specific inference challenges on LLM performance. Integrating LLMs with formal methods is a promising frontier in the Natural Language Processing field, as an important avenue for improving model inference control and explainability. In particular, inductive learning over complex sets of facts and rules, poses unique challenges for current autoregressive models, as they lack explicit symbolic grounding. While they can be complemented by formal systems, the properties delivered by LLMs regarding inductive learning, are not well understood and quantified. Empirical results indicate that the largest LLMs can achieve competitive results against a SOTA Inductive Logic Programming (ILP) system baseline, but also that tracking long predicate relationship chains is a more difficult obstacle than theory complexity for LLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives, or rely on hand-crafted formal grammars specified per domain.
“Existing neurosymbolic approaches either internalize rules through retraining with logic-informed objectives de Souza et al. [2025], or rely on hand-crafted formal grammars specified per domain Pan et al. [2023], Chen et al. [2022, 2023].”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as de Souza et al. [2025]
The citation attaches to the retraining branch of the sentence, but the cited paper describes a different approach. Figure 1 (page unavailable in our copy) says an LLM generates theories and feedback from a logic interpreter is used to refine the prompts iteratively—not to retrain the model with logic-informed objectives. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredLLMs demonstrate strong capabilities in code generation, abductive reasoning, and logic program synthesis.
“Trained on large-scale corpora, LLMs encode broad prior knowledge and demonstrate strong capabilities in code generation, abductive reasoning [6], [7], and logic programs synthesis [8]”
From ADVENT: LLM-Driven Automatic Predicate Invention for ILP (2026) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 19
- Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020. Language models are few-shot learners. In Proceedings of the 34th Interna- tional Conference on Neural Information Processing Sys- tems, NIPS '20. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781713829546.
- Chu, Y.; Cai, S.; and Luo, C. 2023. NuWLS: Improving Lo- cal Search for (Weighted) Partial MaxSAT by New Weight- ing Techniques. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4): 3915-3923.
- Cornelio, C.; and Thost, V. 2021. Synthetic Datasets and Evaluation Tools for Inductive Neural Reasoning. In Pro- ceedings of the 30th International Conference on Inductive Logic Programming, ILP2020-21 @ IJCLR.
- Cropper, A.; and Morel, R. 2021. Learning programs by learning from failures. Machine Learning, 110(4): 801-856.
- Dikovsky, A. J. 1993. On computational complexity of Pro- log programs. Theoretical Computer Science, 119(1): 63- 102.
- Dziri, N.; Lu, X.; Sclar, M.; Li, X. L.; Jian, L.; Lin, B. Y.; West, P.; Bhagavatula, C.; Bras, R. L.; Hwang, J. D.; et al. 2023. Faith and Fate: Limits of Transformers on Composi- tionality. arXiv preprint arXiv:2305.18654. · arXiv
- Huang, J.; and Chang, K. C.-C. 2023. Towards Reasoning in Large Language Models: A Survey. In Rogers, A.; Boyd- Graber, J.; and Okazaki, N., eds., Findings of the Associa- tion for Computational Linguistics: ACL 2023, 1049-1065.
- Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. · arXiv
- Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Welleck, S.; Majumder, B. P.; Gupta, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651. · arXiv
- Nienhuys-Cheng, S.-H.; and de Wolf, R. 1997. What is in- ductive logic programming? Springer.
- Valentino, M.; Dennis, L. A.; and Freitas, A. 2024a. Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement. arXiv:2402.00745. · arXiv
- Quan, X.; Valentino, M.; Dennis, L. A.; and Freitas, A. 2024b. Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving. arXiv:2405.01379. · arXiv
- Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.
- Sinha, K.; Sodhani, S.; Dong, J.; Pineau, J.; and Hamilton, W. L. 2019. CLUTRR: A Diagnostic Benchmark for Induc- tive Reasoning from Text. In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), 4506-4515. Hong Kong, China: Association for Computational Linguistics.
- Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. · arXiv
- Wan, Y.; Wang, W.; Yang, Y.; Yuan, Y.; Huang, J.; He, P.; Jiao, W.; and Lyu, M. R. 2024. A & B == B & A: Trigger- ing Logical Reasoning Failures in Large Language Models. CoRR, abs/2401.00757. · arXiv
- Warren, D. S.; Dahl, V.; Eiter, T.; Hermenegildo, M. V.; Kowalski, R.; and Rossi, F. 2023. Prolog: The Next 50 Years, volume 13900. Springer Nature.
- Yang, F.; Yang, Z.; and Cohen, W. W. 2017. Differentiable learning of logical rules for knowledge base reasoning. Ad- vances in neural information processing systems, 30.
- Zhu, Z.; Xue, Y.; Chen, X.; Zhou, D.; Tang, J.; Schuurmans, D.; and Dai, H. 2024. Large Language Models can Learn Rules. arXiv:2310.07064. · arXiv
Cited by
Sorted by citations
ADVENT: LLM-Driven Automatic Predicate Invention for ILP
Yu, T., Huang, P.-C., Hsu, C., Ku, C.-T., & Kang, Y. (2026) · 2026 IEEE International Conference on Information Reuse and Integration for Data Science (IRI) · doi:10.1109/iri69576.2026.00025
Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming
Chen, Z., & Ma, M. (2026) · arXiv · doi:10.48550/arxiv.2607.23019
- Authors
- De Souza, J. P. G., Carvalho, D., & Freitas, A.
- Affiliations
- University of Manchester (GB)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2025-04-11
- Volume
- 39
- Issue
- 22
- Pages
- 23752-23759
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4409348215
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 1.90
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Cancer Research UK
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (204617, 200021)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, inductive logic programming, LLM evaluation, explainability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
