A review of some techniques for inclusion of domain-knowledge into deep neural networks

“This is done via a modifica-tion to the knowledge-distillation loss proposed by Hinton et al. [65].”

“Understanding models constructed by deep neural networks is an area of intense research activity, and good summaries exist elsewhere [3; 4].”

“It is now accepted that trust comes through understanding of how decisions are made by the machine-constructed models [112], and what are the determining factors in these decisions.”

“Domain-knowledge can also be used to correct biases [114] built into a deep network either declaratively, through the use of constraints, or through the use of loss functions that include “ethical penalty” terms.”

“The computational machinery available in GNN then aggregates and combines the information available in the knowledge graph (an example of this kind of aggregation and pooling of relational information is in [46]).”

“Understanding models constructed by deep neural networks is an area of intense research activity, and good summaries exist elsewhere [3; 4 ] .”

“The work by [77] on Bayesian learning for neural networks also showed how domain-knowledge could help build a prior probability distribution over neural network parameters.”

“In this work, the authors treat an external knowledge graph, such as DBpedia [50], as a source of domain-knowledge allowing entities to be enriched with this knowledge.”

“This form of learning is viewed as an instance of domain-adaptation [34].”

“There are some recent surveys along these lines: [83; 84].”

“[ 70 ] inject domain knowledge at training time via an approach that combines semantic based regularisation and constraint programming [ 71 ] .”

“Techniques for automatic construction of Boolean-valued features from relational domain-knowledge have a long history in the field of ILP [ 15; 16; 17 ] , originating from the LINUS [ 7 ] .”

“One recent example of embedding is demonstrated in [21] where the authors use different text-embedding approaches such as sentence encoder [22] and GPT2 [23] to transform textual domain-knowledge into embedding vectors.”

“The simplification used in vertex-enrichment has been made unnecessary in a recent proposal for transforming the most-specific clause constructed by ILP systems employing mode-directed inverse entailment (MDIE [27]).”

“The intuition is that the nodes (in a problem involving recommender systems, as in [48], a node represents an entity) in the knowledge-graph can be represented with an aggregated neighbourhood information with bias while adopting the central idea of aggregate-and-combine in GNNs.”

“Higher-level domain-concepts are then abduced as compositions of these primitives via a neurally-guided search procedure based on a version of the Bayesian “wake-sleep” algorithm [ 33 ] .”

“In KBANN, the domain knowledge is represented as a set of hierarchically structured propositional rules that directly determines a fixed topological structure of a neural network [86].”

“But it is not just the construction of scientific-assistants that can benefit from this form of man-machine collaboration, and “human-in-the-loop” AI systems are likely to play an increasingly important role in engineering, medicine, healthcare, agriculture, environment and so on [8].”

“…ILBKRME [30] MLP HDNNLR [64] CNN, RNN SBR [68] MLP SBR [69] CNN DL2 [ 66 ] CNN Semantic Loss [ 63 ] CNN LENSR [ 91 ] GNN DANN [ 67 ] MLP PC-LSTM [ 72 ] RNN DomiKnowS [ 73 ] DNN* MultiplexNet [ 74 ] MLP, CNN Analogy Model [ 75 ] RNN Transforming Model KBANN [ 86 ] MLP Cascade-ARTMAP [ 89 ] ARTMAP…”

“…for use Principal Approach Work (Reference) Type of Learner Transforming Data DRM [24; 25] MLP CILP++ [28] MLP R-GCN [46] GNN KGCN [48] GNN KBRD [49] GNN DG-RNN [44] RNN DreamCoder [32] DNN ∗ Gated-K-BERT [38] Transformer VEGNN [5] GNN BotGNN [6] GNN KRISP [45] GNN, Transformer Transforming…”

“There are some recent surveys along these lines: [83; 84].”

“The general intuition behind this idea is that the semantic loss is proportional to a negative logarithm of the probability of generating a state that satis-fies the constraint when sampling values according to some probability distribution.”

“[56] proposed a joint model, called KALE, which embeds facts from knowledge-graphs and logical rules, simultaneously.”

“The optimiser minimises the loss along with the regularisation term resulting in a less complex model in terms of parameters [Kukačka et al., 2017].”

“Learning and inference can then be conducted on these matrices (or tensors) [ 104; 105 ] allowing faster computation.”

“While the above methods use the relational knowledge from a knowledge-graph by altering or adding an attention module within the deep sequence model, a recent method called KRISP [45] introduces such external knowledge at the output (prediction) layer of BERT.”

“DNN based methods arguably work best if the domainknowledge is used to inspire their architecture choices [85].”

“An example of this kind of learning is in [ 82 ] where a BERT model is fine-tuned with data from multiple domains.”

“In a similar line, [67] proposed domain-adapted neural network (DANN) that works with a balanced loss function at the intersection of models based on purely domain-based loss or purely inductive loss.”

“Some recent works are in [47; 48], where a GNN is used for estimation of node importance in a knowledge-graph.”

“[Hamilton et al., 2018] incorporate first-order logic into low dimensional spaces by embedding graphs nodes and represent logical operators as learned geometric relations in the space.”

“Another work that translates domain-knowledge in first-order logic into a deep neural network architecture consisting of the input layer (grounded atoms), propositional layer, quantifier layer and output layer is [ 68 ] .”

“Techniques for automatic construction of Boolean-valued features from relational domainknowledge have a long history in the field of ILP [15; 16; 17], originating from the LINUS [7].”

“Fusing the information in a knowledge-graph in this way into various level of hidden representations in a deep network could also allow quantitative and qualitative assessment of its functioning, leading to knowledge-infused interpretability [42].”

“A similar study is KBCNN [ 87 ] , which first identifies and links domain attributes and concepts consistent with initial domain knowledge.”

“[ 60 ] proposed ordering of embedding space based on rules mined from WordNet and found it to better prior knowledge and generalisation capabilities using these relational embeddings.”

“Techniques for automatic construction of Boolean-valued features from relational domain-knowledge have a long history in the field of ILP [ 15; 16; 17 ] , originating from the LINUS [ 7 ] .”

“…Type of Learner Transforming Data DRM [24; 25] MLP CILP++ [28] MLP R-GCN [46] GNN KGCN [48] GNN KBRD [49] GNN DG-RNN [44] RNN DreamCoder [32] DNN ∗ Gated-K-BERT [38] Transformer VEGNN [5] GNN BotGNN [6] GNN KRISP [45] GNN, Transformer Transforming Loss IPKFL [78] CNN ILBKRME [30] MLP…”

“Transfer learning need not be restricted to deep networks only: in a recent study, [ 80 ] proposes a model that transfers knowledge from a neural network to a decision tree using knowledge distillation framework.”

“If the domain-knowledge expressed in a symbolic form (for example, logical relations that are known to hold in the domain), then the broad area of hybrid neural-symbolic systems (see for example, [9; 10]) is clearly relevant to the material in this paper.”

“CILP++ [28] uses bottom-clauses for data instances to construct feature-vectors for neural networks.”

“…VEGNN [5] GNN BotGNN [6] GNN KRISP [45] GNN, Transformer Transforming Loss IPKFL [78] CNN ILBKRME [30] MLP HDNNLR [64] CNN, RNN SBR [68] MLP SBR [69] CNN DL2 [ 66 ] CNN Semantic Loss [ 63 ] CNN LENSR [ 91 ] GNN DANN [ 67 ] MLP PC-LSTM [ 72 ] RNN DomiKnowS [ 73 ] DNN* MultiplexNet [ 74 ] MLP,…”

“…of these formal representations may continue to be needed for the construction of scientific assistants, the flexibility of natural language may be especially useful in communicating commonsense knowledge to day-to-day machine assistants that need to an informal knowledge of the world [ 110; 111 ] .”

“Techniques for automatic construction of Boolean-valued features from relational domain-knowledge have a long history in the field of ILP [ 15; 16; 17 ] , originating from the LINUS [ 7 ] .”

“Template-based construction of deep network structure can be also seen in Logical Neural Networks (LNNs: [102]).”

“…Approach Work (Reference) Type of Learner Transforming Data DRM [24; 25] MLP CILP++ [28] MLP R-GCN [46] GNN KGCN [48] GNN KBRD [49] GNN DG-RNN [44] RNN DreamCoder [32] DNN ∗ Gated-K-BERT [38] Transformer VEGNN [5] GNN BotGNN [6] GNN KRISP [45] GNN, Transformer Transforming Loss IPKFL [78] CNN…”

“Knowledge graphs have been used successfully in variety of problems arising in information processing domains such as search, recommendation, summarisation [37].”

“However, it requires the use of a language of Lifted Relational Neural Networks (LRNNs) [101].”

“Incorporation of the information in a knowledge-graph into deep neural models–termed “knowledge-infused learning”–is described in [Kursuncu et al., 2019; Sheth et al., 2019].”

“Incorporation of the information in a knowledge-graph into deep neural models–termed “knowledge-infused learning”–is described in [40; 41].”

“[55] was perhaps the first method of regularising embeddings from declarative knowledge encoded in first-order logic.”

“A recent approach called DeepStochLog [94] is a framework that extends the idea of neural predicates in DeepProbLog to defi-nite clause grammars [96].”

“KBANN [Towell and Shavlik, 1994] Propositional logic MLP Cascade-ARTMAP [Tan, 1997] Propositional logic ARTMAP CILP [Avila Garcez and Zaverucha, 1999] Propositional logic RNN LENSR [Xie et al., 2019] Canonical Normal Form Loss function GNN CILP++ [França et al.”

“For instance, in kLog [97] the transformed representation is an undirected bipartite graph in the form of ‘Probabilistic Entity-Relationship model’ [98] which allows the use of a graph-kernel [99] for data classification purpose, where each data instance is represented as a logic program constructed…”

“One challenge that is pointed out in a recent study [74] is that incorporating domain-knowledge in this manner (that is: adding a domain-based loss to the standard problem-specific loss) may not always be suitable while dealing with safety-critical domains where 100% constraint satisfaction is…”

“In [35], for example, a feature-encoder ensures that the feature representation for the target domain that is the same as the one used for the source.”

“There is a long history of propositionalisation: see [18] for a review of some of early use of this technique, and [19; 20] who examine the links between propositionalisation and modern-day use of embeddings in neural networks.”

“The results reported in [5] show significant improvements in the predictive accuracy of GNNs across a large number datasets.”

“Some of the limitations described above could be overcome with the proposal of a hybrid system by Fletcher and Obradovic [ 88 ] .”

“While the precision of these formal representations may continue to be needed for the construction of scientific assistants, the flexibility of natural language may be especially useful in communicating commonsense knowledge to day-to-day machine assistants that need to an informal knowledge of the world [110; 111].”

“Secondly, it can provide concepts that are meaningful for use Principal Approach Work (Reference) Type of Learner Transforming Data DRM [24; 25] MLP CILP++ [28] MLP R-GCN [46] GNN KGCN [48] GNN KBRD [49] GNN DG-RNN [44] RNN DreamCoder [32] DNN ∗ Gated-K-BERT [38] Transformer VEGNN [5] GNN BotGNN…”

“Conceptually, there is no variable-sharing between two or more first-order logic features [25].”

“[Fu, 1995] was one of the earliest works in this domain, in which they proposed a strategy to establish constraints by designating each node in a Hopfield Net to represent a concept and edges to represent their relationships and learn these nets by finding the solution which maximises the greatest number of these cons…”

“A knowledge graph provides a structure representation for knowledge that is accessible to both humans and machines [36].”

“The intuition is that the nodes (in a problem involving recommender systems, as in [48], a node represents an entity) in the knowledge-graph can be represented with an aggregated neighbourhood information with bias while adopting the central idea of aggregate-and-combine in GNNs.”

“…CILP++ [28] MLP R-GCN [46] GNN KGCN [48] GNN KBRD [49] GNN DG-RNN [44] RNN DreamCoder [32] DNN ∗ Gated-K-BERT [38] Transformer VEGNN [5] GNN BotGNN [6] GNN KRISP [45] GNN, Transformer Transforming Loss IPKFL [78] CNN ILBKRME [30] MLP HDNNLR [64] CNN, RNN SBR [68] MLP SBR [69] CNN DL2 [ 66 ] CNN…”

“…Transforming Data DRM [24; 25] MLP CILP++ [28] MLP R-GCN [46] GNN KGCN [48] GNN KBRD [49] GNN DG-RNN [44] RNN DreamCoder [32] DNN ∗ Gated-K-BERT [38] Transformer VEGNN [5] GNN BotGNN [6] GNN KRISP [45] GNN, Transformer Transforming Loss IPKFL [78] CNN ILBKRME [30] MLP HDNNLR [64] CNN, RNN SBR…”

“…CNN Semantic Loss [ 63 ] CNN LENSR [ 91 ] GNN DANN [ 67 ] MLP PC-LSTM [ 72 ] RNN DomiKnowS [ 73 ] DNN* MultiplexNet [ 74 ] MLP, CNN Analogy Model [ 75 ] RNN Transforming Model KBANN [ 86 ] MLP Cascade-ARTMAP [ 89 ] ARTMAP CIL 2 P [ 29 ] RNN DeepProbLog [ 93 ] CNN LRNN [ 101 ] MLP TensorLog [ 105…”

“There is a long history of propositionalisation: see [ 18 ] for a review of some of early use of this technique, and [ 19; 20 ] who examine the links between propositionalisation and modern-day use of embed-dings in neural networks.”

“The sampling technique is refined further in [26].”

“LNNs resemble a tree structure where the leaf nodes represent the facts in the data and background knowledge, the internal nodes implement logical connectives (and, or, implication, etc.) using t -norm operators derived from real-valued logic, and the outputs represent the rules [103].”

“Alibraryfor integrating symbolic domain-knowledge with deep neural networks was introduced recently in [73].”

“The result could be thought of as learning weighted forms of the antecedents present in the rules.”

“[ 70 ] inject domain knowledge at training time via an approach that combines semantic based regularisation and constraint programming [ 71 ] .”

“…[ 89 ] ARTMAP CIL 2 P [ 29 ] RNN DeepProbLog [ 93 ] CNN LRNN [ 101 ] MLP TensorLog [ 105 ] MLP Domain-Aware BERT [ 82 ] Transformer NeuralLog [ 106 ] MLP DeepStochLog [ 94 ] DNN* LNN [ 103 ] MLP Table 1: Some selected works, in no particular order, showing the principal approach of domain…”

“This knowledge can be included in the models by changing the input, loss function, and model architecture [49].”

“From the reference ligands of their original complexes, we extracted initial core structures, which were azabicyclo[2.2.1]heptane, 2-(oxan-3-yloxy)oxane, and benzene, respectively.”

“Early works of including domain knowledge into the loss function date from the mid nineties ( Fu 1995, Dash et al 2022 ) .”