CoChat
Sign Up
Cited by 196Neural Networks and ApplicationsDomain Adaptation and Few-Shot LearningModel Reduction and Neural Networks

The DOI matches its Crossref record; 110 of the 112 references it cites were found in a registry or an index.

A review of some techniques for inclusion of domain-knowledge into deep neural networks

Open access
Dash, Tirtharaj, Chitlangia, Sharad, Ahuja, Aditya, Srinivasan, Ashwin·2022·Scientific Reports
DOI 10.1038/s41598-021-04590-0PMC PMC8776800
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a survey of ways in which existing scientific knowledge are included when constructing models with neural networks. The inclusion of domain-knowledge is of special interest not just to constructing scientific assistants, but also, many other areas that involve understanding data using human-machine collaboration. In many such instances, machine-based model construction may benefit significantly from being provided with human-knowledge of the domain encoded in a sufficiently precise form. This paper examines the inclusion of domain-knowledge by means of changes to: the input, the loss-function, and the architecture of deep networks. The categorisation is for ease of exposition: in practice we expect a combination of such changes will be employed. In each category, we describe techniques that have been shown to yield significant changes in the performance of deep neural networks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-021-04590-0

Cites

110 of 110 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An example of AI for Science. The human-in-the-loop is a biologist. The biologist conducts experiments in a biological system, obtains experimental observations. The biologist then extracts data that can be used to construct machine learning model(s). Additionally, the machine learning system has access to domain knowledge that can be obtained from the biologist. The machine learning system then conveys its explanations to the biologist.
Figure 1 An example of AI for Science. The human-in-the-loop is a biologist. The biologist conducts experiments in a biological system, obtains experimental observations. The biologist then extracts data that can be used to construct machine learning model(s). Additionally, the machine learning system has access to domain knowledge that can be obtained from the biologist. The machine learning system then conveys its explanations to the biologist.CC BY · PubMed Central
Figure 2 The plots from6 showing gains in predictive accuracy of (a) multilayer perceptron (MLP), and (b) graph neural network (GNN) with the inclusion of domain-knowledge. The domain knowledge inclusion method in (a) is a simple technique known as ‘propositionalisation’ 7; and, the method in (b) is a general technique of incorporating domain-knowledge using bottom-graph construction. The results shown are over 70 datasets. No importance to be given to the line joining two points; this is done for visualisation purpose only.
Figure 2 The plots from6 showing gains in predictive accuracy of (a) multilayer perceptron (MLP), and (b) graph neural network (GNN) with the inclusion of domain-knowledge. The domain knowledge inclusion method in (a) is a simple technique known as ‘propositionalisation’ 7; and, the method in (b) is a general technique of incorporating domain-knowledge using bottom-graph construction. The results shown are over 70 datasets. No importance to be given to the line joining two points; this is done for visualisation purpose only.CC BY · PubMed Central
Figure 3 Informal descriptions of (a) logical; and (b) numerical constraints.
Figure 3 Informal descriptions of (a) logical; and (b) numerical constraints.CC BY · PubMed Central
Figure 4 Construction of a deep model M from data (D) using a learner (L). We use π to denote the structure (organisation of various layers, their interconnections etc.) and θ to denote the parameters (synaptic weights) of the deep network. L denotes the loss function (for example, cross-entropy loss in case of classification).
Figure 4 Construction of a deep model M from data (D) using a learner (L). We use π to denote the structure (organisation of various layers, their interconnections etc.) and θ to denote the parameters (synaptic weights) of the deep network. L denotes the loss function (for example, cross-entropy loss in case of classification).CC BY · PubMed Central
Figure 5 Some implications of using domain-knowledge for commonly-used deep network architectures. Although attention-mechanism has also been used recently in many deep network architectures, we mention it only for RNNs and transformers as it is more prominently being used for sequence learning.
Figure 5 Some implications of using domain-knowledge for commonly-used deep network architectures. Although attention-mechanism has also been used recently in many deep network architectures, we mention it only for RNNs and transformers as it is more prominently being used for sequence learning.CC BY · PubMed Central
Figure 6 Introducing background knowledge into deep network by transforming data. T is a transformation block that takes input data D, background knowledge (BK) and outputs transformed data D^(′) that is then used to construct a deep model using a learner L.
Figure 6 Introducing background knowledge into deep network by transforming data. T is a transformation block that takes input data D, background knowledge (BK) and outputs transformed data D^(′) that is then used to construct a deep model using a learner L.CC BY · PubMed Central
Figure 7 Introducing background knowledge into deep network by transforming the loss function L. T block takes an input loss L and outputs a new loss function L^(′) by transforming (augmenting or modifying) L based on background knowledge (BK). The learner L then constructs a deep model using the original data D and the new loss function L^(′).
Figure 7 Introducing background knowledge into deep network by transforming the loss function L. T block takes an input loss L and outputs a new loss function L^(′) by transforming (augmenting or modifying) L based on background knowledge (BK). The learner L then constructs a deep model using the original data D and the new loss function L^(′).CC BY · PubMed Central
Figure 8 Introducing background knowledge into deep network by transforming the model (structure and parameter). In (a), the transformation block T takes a input structure of a model π and outputs a transformed structure π^(′) based on background knowledge (BK). In (b), the transformation block T takes a set of parameters θ of a model and outputs a transformed set of parameters π^(′) based on background knowledge (BK).
Figure 8 Introducing background knowledge into deep network by transforming the model (structure and parameter). In (a), the transformation block T takes a input structure of a model π and outputs a transformed structure π^(′) based on background knowledge (BK). In (b), the transformation block T takes a set of parameters θ of a model and outputs a transformed set of parameters π^(′) based on background knowledge (BK).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 112 references

Research works printed but not linked: 9

  1. Lipton, Z. C. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016). · arXiv
  2. Garcez, A. S. d., Broda, K. B. & Gabbay, D. M. Neural-symbolic learning systems: foundations and applications (Springer, 2012).
  3. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog. 1, 9 (2019).
  4. Ellis, K., Morales, L., Meyer, M. S., Solar-Lezama, A. & Tenenbaum, J. B. Dreamcoder: Bootstrapping domain-specific languages for neurally-guided bayesian program learning. In Proceedings of the 2nd Workshop on Neural Abstract Machines and Program Induction (2018).
  5. Hogan, A. et al. Knowledge graphs. arXiv preprint arXiv:2003.02320 (2020). · arXiv
  6. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 4171–4186 (2019). https://aclweb.org/anthology/papers/N/N19/N19-1423/.
  7. Vishwanathan, S. V. N., Schraudolph, N. N., Kondor, R. & Borgwardt, K. M. Graph kernels. J. Mach. Learn. Res. 11, 1201–1242 (2010).
  8. Sourek, G., Aschenbrenner, V., Zelezny, F., Schockaert, S. & Kuzelka, O. Lifted relational neural networks: Efficient learning of latent relational structures. J. Artif. Intell. Res. 62, 69–100 (2018). · DOI
  9. Anderson, M., Anderson, S. & Armen, C. Medethex: Toward a medical ethics advisor (Caring Machines, In AAAI Fall Symposium, 2005).

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Dash, T., Chitlangia, S., Ahuja, A., & Srinivasan, A.
Affiliations
Birla Institute of Technology and Science, Pilani - Goa Campus (IN)
Corresponding author
Dash, Tirtharaj
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2022-01-20
Volume
12
Issue
1
Pages
1040-1040
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-021-04590-0
PMC
PMC8776800
OpenAlex
W4205511145
Semantic Scholar
8f00fca03482af9c7aaf3c5ff60f8a0f69178935
Cited by
196
References
112
Citations by year
2026: 30 · 2025: 52 · 2024: 52 · 2023: 49 · 2022: 11
FWCI
13.49
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • University of New South Wales
  • Birla Institute of Technology and Science, Pilani
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
Keywords
domain knowledge, deep neural network, loss functions, network architecture, input encoding, human-machine collaboration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review