Learning Explanatory Rules from Noisy Data
Open accessAbstract
Artificial Neural Networks are powerful function approximators capable of modelling solutions to a wide variety of problems, both supervised and unsupervised. As their size and expressivity increases, so too does the variance of the model, yielding a nearly ubiquitous overfitting problem. Although mitigated by a variety of model regularisation methods, the common cure is to seek large amounts of training data--which is not necessarily easily obtained--that sufficiently approximates the data distribution of the domain we wish to test on. In contrast, logic programming methods such as Inductive Logic Programming offer an extremely data-efficient process by which models can be trained to reason on symbolic domains. However, these methods are unable to deal with the variety of domains neural networks can be applied to: they are not robust to noise in or mislabelling of inputs, and perhaps more importantly, cannot be applied to non-symbolic domains where the data is ambiguous, such as operating on raw pixels. In this paper, we propose a Differentiable Inductive Logic framework, which can not only solve tasks which traditional ILP systems are suited for, but shows a robustness to noise and error in the training data which ILP cannot cope with. Furthermore, as it is trained by backpropagation against a likelihood objective, it can be hybridised by connecting it with neural networks over ambiguous data in order to be applied to domains which ILP cannot address, while providing data efficiency and generalisation beyond what neural networks on their own can achieve.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The placeholder approach to predicate invention requires generating all possible invented predicates.
“However, this placeholder approach is limited because it requires that a user manually specify the arity and argument types of a symbol (Law et al., 2014), which rather defeats the point, or requires generating all possible invented predicates (Evans & Grefenstette, 2018; Evans et al., 2021), which is computationally expensive.”
From Inductive Logic Programming At 30: A New Introduction (2022) · cites this paper as Evans & Grefenstette, 2018
Section 3.1 (page unavailable) says the program template specifies the set of invented predicates and the arity of each. Section 3.2 says the system generates clauses satisfying each rule template, subject to restrictions—not all possible invented predicates. Section 7.1 describes exhaustive enumeration of program templates only as a separate, computationally intensive experiment. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured∂ ILP uses neural networks to solve the problem.
“∂ ILP (Evans & Grefenstette, 2018) uses neural networks to solve the problem.”
From Inductive Logic Programming At 30: A New Introduction (2022) · cites this paper as Evans & Grefenstette, 2018
The Introduction and Experiments sections say ∂ILP was connected to a pretrained convolutional neural network to handle ambiguous digit images. Section 4 describes ∂ILP itself as differentiable deduction over weighted logic clauses, so the paper supports neural-network use in that experiment, not as the general mechanism by which ∂ILP solves problems. (LLM, conf=0.82)
Experimental — accuracy not yet measuredSome systems, mostly meta-level approaches, cannot handle infinite domains.
“Some systems, mostly meta-level approaches, cannot handle infinite domains (Corapi et al., 2011; Athakravi et al., 2013; Evans & Grefenstette, 2018; Kaminski et al., 2018; Evans et al., 2021).”
From Inductive Logic Programming At 30: A New Introduction (2022) · cites this paper as Evans & Grefenstette, 2018
Section 2.1 says the paper's system disallows function symbols, and Section 6 (Related Work) says it uses Datalog programs with finite Herbrand universes. This supports the limitation on infinite domains for this system, but the paper does not establish that the systems with this limitation are mostly meta-level approaches. (LLM, conf=0.85)
Experimental — accuracy not yet measuredGenerating all possible invented predicates is computationally expensive.
“However, this placeholder approach is limited because it requires that a user manually specify the arity and argument types of a symbol (Law et al., 2014), which rather defeats the point, or requires generating all possible invented predicates (Evans & Grefenstette, 2018; Evans et al., 2021), which is computationally expensive.”
From Inductive Logic Programming At 30: A New Introduction (2022) · cites this paper as Evans & Grefenstette, 2018
Section 7.1 says that enumerating program templates to avoid hand-engineering required a “massive amount of computation.” Section 3.1 says those templates specify the invented predicates and their arities. The paper therefore supports the computational-cost point, but it does not say its approach generates all possible invented predicates; it describes enumerating templates in a brute-force experiment. (LLM, conf=0.83)
Experimental — accuracy not yet measuredIt is often difficult and expensive to obtain hand-crafted background knowledge.
“However, it is often difficult and expensive to obtain such BK.”
From Inductive Logic Programming At 30: A New Introduction (2022) · cites this paper as Evans & Grefenstette, 2018
Section 5.3.1 describes background knowledge as arithmetic facts supplied for a task, but does not discuss the difficulty or expense of obtaining them. Section 7.1 calls hand-engineering program templates difficult work; those templates are distinct from the background knowledge. Searches for the cost of obtaining background knowledge found no supporting passage. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Meta-interpretive learning of higher-order dyadic datalog: predicate invention revisited
Muggleton, S. H., Lin, D., & Tamaddoni-Nezhad, A. (2015) · Machine Learning · doi:10.1007/s10994-014-5471-y · Cited by 184 · see citing works
Inductive Logic Programming as Abductive Search
Corapi, D., Russo, A., & Lupu, E. (2010) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.iclp.2010.54 · Cited by 60 · see citing works
Meta-Interpretive Learning of Data Transformation Programs
Cropper, A., Tamaddoni-Nezhad, A., & Muggleton, S. H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-40566-7_4 · Cited by 32 · see citing works
TopLog: ILP Using a Logic Program Declarative Bias
Muggleton, S. H., Santos, J. C. A., & Tamaddoni-Nezhad, A. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89982-2_58 · Cited by 29 · see citing works
Flagged first, then by citations
Research works printed but not linked: 42
- Andrychowicz, M., & Kurach, K. (2016). Learning efficient algorithms with hierarchical attentive memory. arXiv preprint arXiv:1602.03218. · arXiv
- Bader, S., Garcez, A. S. d., & Hitzler, P. (2005). Computing first-order logic programs by fibring artificial neural networks.. In FLAIRS Conference, pp. 314-319.
- Bader, S., Hitzler, P., & Hölldobler, S. (2008). Connectionist model generation: A first-order approach. Neurocomputing, 71 (13), 2420-2432.
- Bunel, R. R., Desmaison, A., Mudigonda, P. K., Kohli, P., & Torr, P. (2016). Adaptive neural compilation. In Advances in Neural Information Processing Systems, pp. 1444-1452.
- Corapi, D., Russo, A., & Lupu, E. (2011). Inductive logic programming in answer set programming. In International Conference on Inductive Logic Programming, pp. 91- 97. Springer.
- Cropper, A., & Muggleton, S. H. (2016). Learning higher-order logic programs through abstraction and invention. In Proceedings of the 25th International Joint Conference Artificial Intelligence (IJCAI 2016). IJCAI.
- De Raedt, L., & Kersting, K. (2008). Probabilistic inductive logic programming. In Proba- bilistic Inductive Logic Programming, pp. 1-27. Springer.
- De Raedt, L., Kersting, K., Natarajan, S., & Poole, D. (2016). Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 10 (2), 1-189.
- Eisner, J., Goldlust, E., & Smith, N. A. (2004). Dyna: A declarative language for imple- menting dynamic programs. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, p. 32. Association for Computational Linguistics.
- Esteva, F., & Godo, L. (2001). Monoidal t-norm based logic: towards a logic for left- continuous t-norms. Fuzzy sets and systems, 124 (3), 271-288.
- Feser, J. K., Brockschmidt, M., Gaunt, A. L., & Tarlow, D. (2016). Neural functional programming. arXiv preprint arXiv:1611.01988. · arXiv
- Feser, J. K., Chaudhuri, S., & Dillig, I. (2015). Synthesizing data structure transformations from input-output examples. In ACM SIGPLAN Notices, Vol. 50, pp. 229-239. ACM.
- França, M. V., Zaverucha, G., & Garcez, A. S. d. (2014). Fast relational learning using bottom clause propositionalization with artificial neural networks. Machine learning, 94 (1), 81-104.
- Garcez, A. d., Besold, T. R., de Raedt, L., Földiak, P., Hitzler, P., Icard, T., Kühnberger, K.-U., Lamb, L. C., Miikkulainen, R., & Silver, D. L. (2015). Neural-symbolic learn- ing and reasoning: contributions and challenges. In Proceedings of the AAAI Spring Symposium on Knowledge Representation and Reasoning: Integrating Symbolic and Neural Approaches, Stanford.
- Garcez, A. S. d., Gabbay, D. M., & Lamb, L. C. (2014). A neural cognitive model of argumentation with application to legal inference and decision making. Journal of Applied Logic, 12 (2), 109-127.
- Garcez, A. S. d., Gabbay, D. M., Ray, O., & Woods, J. (2007). Abductive reasoning in neural-symbolic systems. Topoi, 26 (1), 37-49.
- Garcez, A. S., Lamb, L. C., & Gabbay, D. M. (2008). Neural-symbolic cognitive reasoning. Springer Science & Business Media.
- Gaunt, A. L., Brockschmidt, M., Singh, R., Kushman, N., Kohli, P., Taylor, J., & Tarlow, D. (2016). Terpret: A probabilistic programming language for program induction. arXiv preprint arXiv:1608.04428. · arXiv
- Graves, A., Wayne, G., & Danihelka, I. (2014). Neural turing machines. arXiv preprint arXiv:1410.5401. · arXiv
- Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626), 471-476.
- Grefenstette, E., Hermann, K. M., Suleyman, M., & Blunsom, P. (2015). Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems, pp. 1828-1836.
- Guillame-Bert, M., Broda, K., & Garcez, A. d. (2010). First-order logic learning in artificial neural networks. In The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE.
- Hölldobler, S., & Kalinke, Y. (1991). Towards a new massively parallel computational model for logic programming. In In ECAI'94 workshop on Combining Symbolic and Connectioninst Processing. Citeseer.
- Hölldobler, S., Kalinke, Y., & Störr, H.-P. (1999). Approximating the semantics of logic programs by recurrent neural networks. Applied Intelligence, 11 (1), 45-58.
- Joulin, A., & Mikolov, T. (2015). Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in neural information processing systems, pp. 190-198.
- Kaiser, L. (2015). Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228. · arXiv
- Khot, T., Natarajan, S., Kersting, K., & Shavlik, J. (2015). Gradient-based boosting for sta- tistical relational learning: the markov logic network and missing data cases. Machine Learning, 100 (1), 75-100.
- Landwehr, N., Kersting, K., & De Raedt, L. (2005). nfoil: Integrating naıve bayes and foil. In Proceedings of the twentieth national conference on artificial intelligence (AAAI-05), pp. 795-800.
- LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 (11), 2278-2324.
- McCarthy, J. (1988). Epistemological challenges for connectionism. Behavioral and Brain Sciences, 11 (01), 44-44.
- Muggleton, S. (1995). Inverse entailment and progol. New generation computing, 13 (3-4), 245-286.
- Muggleton, S., & De Raedt, L. (1994). Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19, 629-679.
- Muggleton, S. H., Lin, D., Pahlavi, N., & Tamaddoni-Nezhad, A. (2014). Meta-interpretive learning: application to grammatical inference. Machine Learning, 94 (1), 25-49.
- Natarajan, S., Khot, T., Kersting, K., Gutmann, B., & Shavlik, J. (2012). Gradient-based boosting for statistical relational learning: The relational dependency network case. Machine Learning, 86 (1), 25-56.
- Neelakantan, A., Le, Q. V., & Sutskever, I. (2015). Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv:1511.04834. · arXiv
- Quinlan, J. R., & Cameron-Jones, R. M. (1995). Induction of logic programs: Foil and related systems. New Generation Computing, 13 (3-4), 287-312.
- Reed, S., & de Freitas, N. (2015). Neural programmer-interpreters. arXiv preprint arXiv:1511.06279. · arXiv
- Riedel, S., Bošnjak, M., & Rocktäschel, T. (2016). Programming with a differentiable forth interpreter. arXiv preprint arXiv:1605.06640. · arXiv
- Serafini, L., & Garcez, A. (2016). Logic tensor networks: Deep learning and logical reasoning from data and knowledge. arXiv preprint arXiv:1606.04422. · arXiv
- Steinbach, B., & Kohut, R. (2002). Neural networks-a model of boolean functions. In Boolean Problems, Proceedings of the 5th International Workshop on Boolean Prob- lems, pp. 223-240.
- Tamaddoni-Nezhad, A., & Muggleton, S. (2009). The lattice structure and refinement operators for the hypothesis space bounded by a bottom clause. Machine Learning, 76 (1), 37-72.
- Wang, W. Y., Mazaitis, K., & Cohen, W. W. (2015). A soft version of predicate invention based on structured sparsity.. In IJCAI, pp. 3918-3924.
Cited by
Sorted by citations
A review of some techniques for inclusion of domain-knowledge into deep neural networks
Dash, T., Chitlangia, S., Ahuja, A., & Srinivasan, A. (2022) · Scientific Reports · doi:10.1038/s41598-021-04590-0 · Cited by 196 · see citing works
Learning programs by learning from failures
Cropper, A., & Morel, R. (2021) · Machine Learning · doi:10.1007/s10994-020-05934-z · Cited by 86 · see citing works
Inductive Logic Programming At 30: A New Introduction
Cropper, A., & Dumančić, S. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13507 · Cited by 83 · see citing works
Abduction-Based Explanations for Machine Learning Models
Ignatiev, A., Narodytska, N., & Marques-Silva, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011511 · Cited by 40 · see citing works
Natural Language Generation Using Sequential Models: A Survey
Pandey, A. K., & Roy, S. S. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11281-6 · Cited by 29 · see citing works
Logical reduction of metarules
Cropper, A., & Tourret, S. (2020) · Machine Learning · doi:10.1007/s10994-019-05834-x · Cited by 29 · see citing works
Learning Relational Representations with Auto-encoding Logic Programs
Dumancic, S., Guns, T., Meert, W., & Blockeel, H. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/842 · Cited by 27 · see citing works
Forgetting to Learn Logic Programs
Cropper, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5776
- Authors
- Evans, R., & Grefenstette, E.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- ORCID
- Grefenstette, Edward
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2018-01-26
- Volume
- 61
- Pages
- 1-64
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2770392471
- Cited by
- 426
- References
- 6
- Citations by year
- 2026: 28 · 2025: 35 · 2024: 46 · 2023: 54 · 2022: 50 · 2021: 72 · 2020: 71 · 2019: 47 · 2018: 21 · 2017: 1
- FWCI
- 29.53
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- inductive logic programming, noisy training data, noise robustness, hybrid neural-symbolic models, data efficiency, overfitting, ambiguous data, generalization, symbolic reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
