Knowledge Base Completion: Baselines Strike Back
Open accessAbstract
Many papers have been published on the knowledge base completion task in the past few years.Most of these introduce novel architectures for relation learning that are evaluated on standard datasets such as FB15k and WN18.This paper shows that the accuracy of almost all models published on the FB15k can be outperformed by an appropriately tuned baseline -our reimplementation of the Dist-Mult model.Our findings cast doubt on the claim that the performance improvements of recent models are due to architectural changes as opposed to hyperparameter tuning or different training objectives.This should prompt future research to re-consider how the performance of models is evaluated and reported.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When carefully tuned, DistMult can still reach state-of-the-art performance.
“Despite this flaw, it has been demonstrated by Kadlec et al . [26] that, when carefully tuned, DistMult can still reach state-of-the-art performance.”
From Knowledge Graph Embedding for Link Prediction (2021) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Complex Embeddings for Simple Link Prediction
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016) · arXiv · doi:10.48550/arxiv.1606.06357 · Cited by 2,100 · see citing works
Flagged first, then by citations
Research works printed but not linked: 36
- Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Free- base: A collaboratively created graph database for structuring human knowledge. In Proceed- ings of the 2008 ACM SIGMOD International Conference on Management of Data. ACM, New York, NY, USA, SIGMOD '08, pages 1247-1250. https://doi.org/10.1145/1376616.1376746. · DOI · Paper page
- Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2014. A semantic matching energy function for learning with multi-relational data. Ma- chine Learning 94(2):233-259.
- Antoine Bordes, Nicolas Usunier, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embed- dings for Modeling Multi-Relational Data. NIPS 26:2787-2795. https://doi.org/10.1007/s13398- 014-0173-7. · DOI
- Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embed- dings of knowledge bases. In Conference on artifi- cial intelligence. EPFL-CONF-192344.
- Kenneth Ward Church. 2017. Emerging trends: I did it, I did it, I did it, but... Natu- ral Language Engineering 23(03):473-480. https://doi.org/10.1017/S1351324917000067. · DOI
- Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew Mccallum. 2017. Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks. EACL .
- Christiane Fellbaum. 1998. WordNet. Wiley Online Library.
- Alberto García-Durán, Antoine Bordes, and Nico- las Usunier. 2015. Composing Relationships with Translations. In Conference on Empir- ical Methods in Natural Language Processing (EMNLP 2015). Lisbonne, Portugal, pages 286-290. https://doi.org/10.18653/v1/D15-1034. · DOI
- Alberto Garcia-Duran, Antoine Bordes, Nicolas Usunier, and Yves Grandvalet. 2016. Combin- ing Two And Three-Way Embeddings Models for Link Prediction in Knowledge Bases. Journal of Artificial Intelligence Research 55:715--742. https://doi.org/10.1613/jair.5013. · DOI
- Katsuhiko Hayashi and Masashi Shimbo. 2017. On the Equivalence of Holographic and Com- plex Embeddings for Link Prediction pages 1-8. http://arxiv.org/abs/1702.05563. · arXiv
- Shizhu He, Kang Liu, Guoliang Ji, and Jun Zhao. 2015. Learning to Represent Knowledge Graphs with Gaussian Embedding. CIKM '15 Proceedings of the 24th ACM International on Conference on In- formation and Knowledge Management pages 623- 632. https://doi.org/10.1145/2806416.2806502. · DOI
- Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge Graph Embedding via Dynamic Mapping Matrix. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers) pages 687-696. http://www.aclweb.org/anthology/P15-1067.
- Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2016. Knowledge Graph Completion with Adap- tive Sparse Transfer Matrix. Proceedings of the 30th Conference on Artificial Intelligence (AAAI 2016) pages 985-991.
- Minlie Huang Yang Yang Jun Feng and Xiaoyan Zhu. 2015. GAKE: Graph Aware Knowledge Emeb- dding. In Proceedings of the 27th International Conference on Computational Linguistics (COL- ING'16). pages 641-651.
- Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: a Method for Stochastic Optimization. International Conference on Learning Representations pages 1- 13.
- Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the Association for Computational Linguistics 3:211-225. https://doi.org/10.1186/1472-6947-15- S2-S2. · DOI
- Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2015a. Modeling relation paths for representation learn- ing of knowledge bases. CoRR abs/1506.00379. http://arxiv.org/abs/1506.00379. · arXiv
- Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015b. Learning Entity and Re- lation Embeddings for Knowledge Graph Comple- tion. Proceedings of the Twenty-Ninth AAAI Confer- ence on Artificial Intelligence Learning pages 2181- 2187.
- Qiao Liu, Liuyi Jiang, Minghao Han, Yao Liu, and Zhiguang Qin. 2016. Hierarchical random walk in- ference in knowledge graphs. In Proceedings of the 39th International ACM SIGIR conference on Re- search and Development in Information Retrieval. ACM, pages 445-454.
- Dat Quoc Nguyen. 2017. An overview of embedding models of entities and rela- tionships for knowledge base completion https://arxiv.org/pdf/1703.08098.pdf. · arXiv
- Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. 2016. STransE: a novel embedding model of entities and relationships in knowledge bases. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies pages 460-466. https://doi.org/10.18653/v1/N16- 1054. · DOI · Paper page
- Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A Review of Relational Machine Learning for Knowledge Graph. Proceedings of the IEEE (28):1-23. https://doi.org/10.1109/JPROC.2015.2483592. · DOI
- Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic Embeddings of Knowledge Graphs. AAAI pages 1955-1961. http://arxiv.org/abs/1510.04935. · arXiv
- Mathias Niepert. 2016. Discriminative gaifman mod- els. In Advances in Neural Information Processing Systems. pages 3405-3413.
- Martin Riedmiller and Heinrich Braun. 1993. A direct adaptive method for faster backpropagation learn- ing: The rprop algorithm. In Neural Networks, 1993., IEEE International Conference on. IEEE, pages 586-591.
- Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling Relational Data with Graph Convo- lutional Networks http://arxiv.org/abs/1703.06103. · arXiv
- Yelong Shen, Po-Sen Huang, Ming-Wei Chang, and Jianfeng Gao. 2016. Implicit reasonet: Model- ing large-scale structured relationships with shared memory. arXiv preprint arXiv:1611.04642 . · arXiv
- Baoxu Shi and Tim Weniger. 2017. ProjE : Embedding Projection for Knowledge Graph Completion. AAAI .
- Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. Proceedings of the Advances in Neural Information Processing Systems 26 (NIPS 2013) .
- Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. Proceedings of the 3rd Workshop on Con- tinuous Vector Space Models and their Composition- ality pages 57-66.
- Théo Trouillon, Christopher R. Dance, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2017. Knowledge Graph Completion via Complex Tensor Factorization http://arxiv.org/abs/1702.06879. · arXiv
- Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. AAAI Conference on Artificial Intelligence pages 1112-1119.
- Zhigang Wang and Juanzi Li. 2016. Text-enhanced representation learning for knowledge graph. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. AAAI Press, pages 1293-1299.
- Han Xiao, Minlie Huang, and Xiaoyan Zhu. 2017. Ssp: Semantic space projection for knowledge graph em- bedding with text descriptions. In Pro-ceedings of the 31st AAAI Conference on Artificial In-telligence.
- Bishan Yang, Wen-tau Yih, Xiaodong He, Jian- feng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Infer- ence in Knowledge Bases. ICLR page 12. http://arxiv.org/abs/1412.6575. · arXiv
- Hee-geun Yoon, Hyun-je Song, Seong-bae Park, and Se-young Park. 2016. A Translation-Based Knowl- edge Graph Embedding Preserving Logical Property of Relations. Naacl pages 1-9.
Cited by
Sorted by citations
Knowledge Graph Embedding for Link Prediction
Rossi, A., Barbosa, D., Firmani, D., Matinata, A., & Merialdo, P. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3424672 · Cited by 601 · see citing works
“[26] demonstrate that a carefully tuned implementation of DistMult [71] can achieve state-of-the-art performances, surpassing most of its own successors, raising questions on whether we are developing beer LP models or just tuning beer hyperparameters.”
BackgroundInfluentialSimplE Embedding for Link Prediction in Knowledge Graphs
Kazemi, S. M., & Poole, D. (2018) · arXiv · doi:10.48550/arxiv.1802.04868 · Cited by 476 · see citing works
“In future, SimplE can be improved or may help improve relational learning in several ways including: 1- building ensembles of SimplE models as (Kadlec et al., 2017) do it for DistMult, 2- adding SimplE to the relationlevel ensembles of Wang et al.”
MethodSemantic embeddings of generic objects for zero-shot learning
Hascoet, T., Ariki, Y., & Takiguchi, T. (2019) · EURASIP Journal on Image and Video Processing · doi:10.1186/s13640-018-0371-x · Cited by 9 · see citing works
- Authors
- Kadlec, R., Bajgar, O., & Kleindienst, J.
- Affiliations
- IBM (United States) (US)
- ORCID
- Kadlec, Rudolf
- Venue
- Proceedings of the 2nd Workshop on Representation Learning for NLP
- Published
- 2017
- Pages
- 69-74
- OpenAlex
- W2619583282
- Cited by
- 173
- References
- 2
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 5 · 2023: 16 · 2022: 21 · 2021: 27 · 2020: 42 · 2019: 31 · 2018: 17 · 2017: 6
- FWCI
- 11.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Representation Learning for NLP (RepL4NLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Keywords
- knowledge base completion, relation learning, DistMult, hyperparameter optimization, training objectives, model evaluation, baseline model
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
