Embedding Multimodal Relational Data for Knowledge Base Completion
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Representing entities and relations in an embedding space is a well-studied approach for machine learning on relational data.Existing approaches, however, primarily focus on simple link structure between a finite set of entities, ignoring the variety of data types that are often used in knowledge bases, such as text, images, and numerical values.In this paper, we propose multimodal knowledge base embeddings (MKBE) that use different neural encoders for this variety of observed data, and combine them with existing relational models to learn embeddings of the entities and multimodal data.Further, using these learned embedings and different neural decoders, we introduce a novel multimodal imputation model to generate missing multimodal values, like text and images, from information in the knowledge base.We enrich existing relational datasets to create two novel benchmarks that contain additional information such as textual descriptions and images of the original entities.We demonstrate that our models utilize this additional information effectively to provide more accurate link prediction, achieving state-of-the-art results with a considerable gap of 5-7% over existing methods.Further, we evaluate the quality of our generated multimodal values via a user study.We have release the datasets and the opensource implementation of our models at https: //github.com/pouyapez/mkbe.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 2×
“, 2017) structure with pix2pix-GAN (Isola et al., 2017) model to generate high-quality images, conditioning the generator on the entity embeddings in the knowledge base representation.”
BackgroundMethodThe MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 6,627
Modeling Relational Data with Graph Convolutional Networks
Schlichtkrull, M., Kipf, T. N., Bloem, P., van den Berg, R., Titov, I., & Welling, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93417-4_38 · Cited by 5,570
Knowledge Graph Embedding by Translating on Hyperplanes
Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8870 · Cited by 3,915 · cited 2×
“…they use matrix and tensor multiplication (Nickel et al., 2011; Yang et al., 2015; Socher et al., 2013), Euclidean distance (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015), circular correlation (Nickel et al., 2016), or the Hermitian dot product (Trouillon et al., 2016) as scoring…”
BackgroundMethodLearning Entity and Relation Embeddings for Knowledge Graph Completion
Lin, Y., Liu, Z., Sun, M., Liu, Y., & Zhu, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9491 · Cited by 3,749 · cited 2×
“…use matrix and tensor multiplication (Nickel et al., 2011; Yang et al., 2015; Socher et al., 2013), Euclidean distance (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015), circular correlation (Nickel et al., 2016), or the Hermitian dot product (Trouillon et al., 2016) as scoring function.”
BackgroundConvolutional 2D Knowledge Graph Embeddings
Dettmers, T., Minervini, P., Stenetorp, P., & Riedel, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11573 · Cited by 2,486
Self-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267 · cited 2×
“Furthermore, for the case that the object entity is categorical, we embed it through a dense layer with a recently introduced selu activation (Klambauer et al., 2017), with the same number of nodes as the”
MethodEmbedding Entities and Relations for Learning and Inference in Knowledge Bases
Yang, B., Yih, W.-T., He, X., Gao, J., & Deng, L. (2014) · arXiv · doi:10.48550/arxiv.1412.6575 · Cited by 2,252 · cited 9×
“In particular, they use matrix and tensor multiplication (Nickel et al., 2011; Yang et al., 2015; Socher et al., 2013), Euclidean distance (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015), circular correlation (Nickel et al., 2016), or the Hermitian dot product (Trouillon et al., 2016) as…”
BackgroundMethodInfluentialComplex Embeddings for Simple Link Prediction
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016) · arXiv · doi:10.48550/arxiv.1606.06357 · Cited by 2,100
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Cited by
Knowledge Graph Embedding for Link Prediction
Rossi, A., Barbosa, D., Firmani, D., Matinata, A., & Merialdo, P. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3424672 · Cited by 601
“For instance, MKBE [55] uses separate neural encoders for handling text, images, and numerical values, and combines them with structure-based LP models; the work of Trisedya et al.”
MethodA Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal
Liang, K., Meng, L., Liu, M., Liu, Y., Tu, W., Wang, S., Zhou, S., Liu, X., Sun, F., & He, K. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3417451 · Cited by 553
Multi-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 291
Multi-modal Knowledge Graphs for Recommender Systems
Sun, R., Cao, X., Zhao, Y., Wan, J., Zhou, K., Zhang, F., Wang, Z., & Zheng, K. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411947 · Cited by 265
MMKG: Multi-modal Knowledge Graphs
Liu, Y., Li, H., Garcia-Duran, A., Niepert, M., Onoro-Rubio, D., & Rosenblum, D. S. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-21348-0_30 · Cited by 257
Heinzerling, B., & Inui, K. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.153 · Cited by 78
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56
Metadata
- Authors
- Pezeshkpour, P., Chen, L., & Singh, S.
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 3208-3218
- OpenAlex
- W2891850145
- Cited by
- 145
- References
- 14
- FWCI
- 4.79
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.