CoChat
Sign Up
PreprintCited by 2,252Advanced Graph Neural NetworksLogic, Reasoning, and KnowledgeSemantic Web and Ontologies

Embedding Entities and Relations for Learning and Inference in Knowledge Bases

Open access
Yang, Bishan, Yih, Wen-tau, He, Xiaodong, Gao, Jianfeng, Deng, Li·2014·arXiv
DOI 10.48550/arxiv.1412.6575arXiv 1412.6575
Open this paper in CoChat — free accountOpen PDF

Abstract

We consider learning representations of entities and relations in KBs using the neural-embedding approach. We show that most existing models, including NTN (Socher et al., 2013) and TransE (Bordes et al., 2013b), can be generalized under a unified learning framework, where entities are low-dimensional vectors learned from a neural network and relations are bilinear and/or linear mapping functions. Under this framework, we compare a variety of embedding models on the link prediction task. We show that a simple bilinear formulation achieves new state-of-the-art results for the task (achieving a top-10 accuracy of 73.2% vs. 54.7% by TransE on Freebase). Furthermore, we introduce a novel approach that utilizes the learned relation embeddings to mine logical rules such as "BornInCity(a,b) and CityInCountry(b,c) => Nationality(a,c)". We find that embeddings learned from the bilinear objective are particularly good at capturing relational semantics and that the composition of relations is characterized by matrix multiplication. More interestingly, we demonstrate that our embedding-based rule extraction approach successfully outperforms a state-of-the-art confidence-based rule mining approach in mining Horn rules that involve compositional reasoning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1412.6575

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The second group of multiplicative models includes DistMult and Complex, which can outperform additive models by capturing more semantic information.

    “The second group of multiplicative models includes DistMult [20] and Complex [21], which can outperform the additive models by capturing more semantic information [19].”

    From A Survey on Knowledge Graph Embeddings for Link Prediction (2021) · cites this paper as [20]

    The Results (p. 5) report that DistMult outperformed the additive DistAdd model on the evaluated datasets. The Introduction (p. 1) says bilinear embeddings capture the compositional semantics of relations, but the paper does not evaluate Complex or establish that both named models outperform additive models by capturing more semantic information. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Traditional semantic-based models focus only on the information of the triple itself and do not fuse any additional information.

    “The traditional semantic-based models focus only on the information of the triple itself and do not fuse any additional information, as DistMult [20] and Complex [21] do.”

    From A Survey on Knowledge Graph Embeddings for Link Prediction (2021) · cites this paper as [20]

  • DistMult can only model symmetric relations.

    “Since DistMult does not distinguish between head and tail entities, it can only model symmetric relations.”

    From SimplE Embedding for Link Prediction in Knowledge Graphs (2018) · cites this paper as [46]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 32

  1. Auer, S¨ oren, Bizer, Christian, Kobilarov, Georgi, Lehmann, Jens, Cyganiak, Richard, and Ives, Zachary. Dbpedia: A nucleus for a web of open data. In The semantic web , pp. 722-735. Springer, 2007.
  2. Bordes, Antoine, Weston, Jason, Collobert, Ronan, and Bengio, Yoshua. Learning structured embeddings of knowledge bases. In AAAI , 2011.
  3. Bordes, Antoine, Glorot, Xavier, Weston, Jason, and Bengio, Yoshua. A semantic matching energy function for learning with multi-relational data. Machine Learning , pp. 1-27, 2013a.
  4. Bordes, Antoine, Usunier, Nicolas, Garcia-Duran, Alberto, Weston, Jason, and Yakhnenko, Oksana. Translating embeddings for modeling multi-relational data. In NIPS , 2013b.
  5. Bowman, Samuel R. Can recursive neural tensor networks learn logical reasoning? In ICLR , 2014.
  6. Chang, Kai-Wei, Yih, Wen-tau, Yang, Bishan, and Meek, Chris. Typed tensor decomposition of knowledge bases for relation extraction. In EMNLP , 2014.
  7. Deng, Li, Hinton, G., and Kingsbury, B. New types of deep neural network learning for speech recognition and related applications: An overview. In in ICASSP , 2013.
  8. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research , 12:2121-2159, 2011.
  9. Gao, Jianfeng, Pantel, Patrick, Gamon, Michael, He, Xiaodong, Deng, Li, and Shen, Yelong. Modeling interestingness with deep neural networks. In EMNLP , 2014.
  10. Garc´ ıa-Dur´ an, Alberto, Bordes, Antoine, and Usunier, Nicolas. Effective blending of two and threeway interactions for modeling multi-relational data. In Machine Learning and Knowledge Discovery in Databases , pp. 434-449. Springer, 2014.
  11. Getoor, Lise and Taskar, Ben (eds.). Introduction to Statistical Relational Learning . The MIT Press, 2007.
  12. Hinton, Geoff, Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. Deep neural networks for acoustic modeling in speech recognition. IEEE Sig. Proc. Mag. , 29:82-97, 2012.
  13. Huang, Po-Sen, He, Xiaodong, Gao, Jianfeng, Deng, Li, Acero, Alex, and Heck, Larry. Learning deep structured semantic models for Web search using clickthrough data. In CIKM , 2013.
  14. Hutchinson, B, Deng, L., and Yu, D. Tensor deep stacking networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1944-1957, 2013.
  15. Jenatton, Rodolphe, Le Roux, Nicolas, Bordes, Antoine, and Obozinski, Guillaume. A latent factor model for highly multi-relational data. In NIPS , 2012.
  16. Kemp, Charles, Tenenbaum, Joshua B, Griffiths, Thomas L, Yamada, Takeshi, and Ueda, Naonori. Learning systems of concepts with an infinite relational model. In AAAI , volume 3, pp. 5, 2006.
  17. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In NIPS , pp. 3111-3119, 2013.
  18. Nickel, Maximilian, Tresp, Volker, and Kriegel, Hans-Peter. A three-way model for collective learning on multi-relational data. In ICML , pp. 809-816, 2011.
  19. Nickel, Maximilian, Tresp, Volker, and Kriegel, Hans-Peter. Factorizing YAGO: scalable machine learning for linked data. In WWW , pp. 271-280, 2012.
  20. Paccanaro, Alberto and Hinton, Geoffrey E. Learning distributed representations of concepts using linear relational embedding. IEEE Transactions on Knowledge and Data Engineering , 13(2): 232-244, 2001.
  21. Richardson, Matthew and Domingos, Pedro. Markov logic networks. Machine learning , 62(1-2): 107-136, 2006.
  22. Rockt¨ aschel, Tim, Boˇ snjak, Matko, Singh, Sameer, and Riedel, Sebastian. Low-dimensional embeddings of logic. In ACL Workshop on Semantic Parsing , 2014.
  23. Schoenmackers, Stefan, Etzioni, Oren, Weld, Daniel S, and Davis, Jesse. Learning first-order horn clauses from web text. In EMNLP , 2010.
  24. Shen, Yelong, He, Xiaodong, Gao, Jianfeng, Deng, Li, and Mesnil, Gregoire. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM , 2014a.
  25. Shen, Yelong, He, Xiaodong, Gao, Jianfeng, Deng, Li, and Mesnil, Gr´ egoire. Learning semantic representations using convolutional neural networks for Web search. In WWW , pp. 373-374, 2014b.
  26. Singh, Ajit P and Gordon, Geoffrey J. Relational learning via collective matrix factorization. In KDD , pp. 650-658. ACM, 2008.
  27. Socher, Richard, Huval, Brody, Manning, Christopher D., and Ng, Andrew Y. Semantic compositionality through recursive matrix-vector spaces. In EMNLP-CoNLL , 2012.
  28. Socher, Richard, Chen, Danqi, Manning, Christopher D., and Ng, Andrew Y. Reasoning with neural tensor networks for knowledge base completion. In NIPS , 2013.
  29. Sutskever, Ilya, Tenenbaum, Joshua B, and Salakhutdinov, Ruslan. Modelling relational data using Bayesian clustered tensor factorization. In NIPS , pp. 1821-1828, 2009.
  30. Vinyals, O., Jia, Y ., Deng, L., and Darrell, T. Learning with recursive perceptual representations. In NIPS , 2012.
  31. Yih, Wen-tau, He, Xiaodong, and Meek, Christopher. Semantic parsing for single-relation question answering. In ACL , 2014.
  32. Yu, D., Deng, L., and Seide, F. The deep tensor neural network with applications to large vocabulary speech recognition. IEEE Trans. Audio, Speech and Language Proc. , 21(2):388 -396, 2013.

Cited by

Sorted by citations

Authors
Yang, B., Yih, W.-T., He, X., Gao, J., & Deng, L.
Affiliations
  • Cornell University (US)
  • Microsoft (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2014
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1412.6575
arXiv
1412.6575
OpenAlex
W1533230146
Cited by
2,252
References
0
Citations by year
2026: 17 · 2025: 169 · 2024: 352 · 2023: 331 · 2022: 322 · 2021: 396 · 2020: 290 · 2019: 192 · 2018: 94 · 2017: 46 · 2016: 33 · 2015: 9 · 2012: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
Keywords
knowledge graphs, first-order logic rules, inductive logic programming, knowledge graph completion, fact-checking, rule-based reasoning, knowledge base inference

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review