Knowledge Graph Embedding for Link Prediction

In the paper

“Despite this flaw, it has been demonstrated by Kadlec et al . [26] that, when carefully tuned, DistMult can still reach state-of-the-art performance.”

What the sources say

• Supported

  The Introduction says that DistMult with proper hyperparameters and a training objective outperformed 27 of 29 models on the standard Hits@10 metric across WN18 and FB15k. The Model section also reports a best mean reciprocal rank on FB15k, while noting weaker results on some WN18 metrics. This supports the qualified claim that careful tuning can yield state-of-the-art performance, not that DistMult leads on every metric. (LLM, conf=0.92)

  cites [26] [26] R. Kadlec, O. Bajgar, and J. Kleindienst. Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744 , 2017. - source

In the paper

“ComplEx [61], similarly to DistMult, forces each relation embedding to be a diagonal matrix, but extends such formulation in the complex space: 𝒉 ∈ C 𝑑 , 𝒕 ∈ C 𝑑 , 𝒓 ∈ C 𝑑 × 𝑑 . In the complex space, the bilinear product becomes a Hermitian product, where in lieu of the traditional 𝒕 , its conjugate-transpose 𝒕 is employed. This disables the commutativeness above mentioned for the scoring function”

What the sources say

• Supported

  The model comparison (Section 2.1, p. 3) gives DistMult a real-valued scoring function and ComplEx a complex-valued one with a conjugated entity embedding. The Introduction (p. 2) says this makes the score order-dependent and enables antisymmetric relations; Section 3 (p. 4) explicitly describes the formulation as DistMult with real embeddings extended to handle asymmetry through conjugation. (LLM, conf=0.94)

  cites [61] [61] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , pages 2071-2080, 2016. - source

In the paper

“STransE [42], in addition to the 𝑑 -sized embeddings seen in TransE, associates to each relation 𝑟 two additional 𝑑 × 𝑑 independent matrices 𝑾 𝒉 𝒓 and 𝑾 𝒕 𝒓 . When computing the score of a fact ⟨ ℎ, 𝑟, 𝑡 ⟩ , before operating the usual translation, 𝒉 is pre-multiplied by 𝑾 𝒉 𝒓 and 𝒕 by 𝑾 𝒕 𝒓 ; this amounts to use relation-specific embeddings for the head and tail, alleviating the issues suffered by”

What the sources say

• Supported

  Section 2 (“Our approach”; page unavailable) defines the STransE score using separate relation-specific matrices applied to the head and tail before the translation. The results discussion (page unavailable) says its full matrices help it model 1-to-many, many-to-one, and especially many-to-many relations better than comparable models. (LLM, conf=0.92)

  cites [42] [42] D. Q. Nguyen, K. Sirts, L. Qu, and M. Johnson. Stranse: a novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08140 , 2016. - source

In the paper

“They demonstrate the severity of said leakage by showing that a simple rule-based model based on inverse relation detection, dubbed Inverse Model, achieves state-of-the-art results in both WN18 and FB15k.”

What the sources say

• Supported

  The Inverse Model section (p. 5) says the authors construct a simple rule-based model that detects inverse relationships and call it the inverse model. The Results section (p. 5) says it achieves state-of-the-art results on many metrics for both FB15k and WN18, demonstrating the severity of inverse-relation leakage. (LLM, conf=0.95)

  cites [11] [11] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018. - source

In the paper

“FB15k-237 was built to be a more challenging dataset: the authors first selected facts from FB15k involving the 401 largest relations and removed all equivalent or inverse relations.”

What the sources say

• Supported

  Section 4.2 describes a harder dataset derived from FB15K, which the paper calls FB15KSelected. Section 4.3 says it began with the 401 most frequent FB15K relations and retained only one from each detected set of inverse or near-duplicate relations, leaving 237. (LLM, conf=0.88)

  cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - source

In the paper

“RotatE [55] represents relations as rotations in a complex latent space, with 𝒉 , 𝒓 and 𝒕 all belonging to C 𝑑 . The 𝒓 embedding is a rotation vector: in all its elements, the complex component conveys the rotation along that axis, whereas the real component is always equal to 1. The rotation 𝒓 is applied to 𝒉 by operating an element-wise product (once again noted with ⊙ in 1). L1 norm is used for”

What the sources say

• Supported

  Section 3.2 (p. 4) defines relations as element-wise rotations in complex vector space. Its three lemmas state that RotatE can infer symmetry/antisymmetry, inversion, and composition patterns. (LLM, conf=0.94)

  cites [55] [55] Z. Sun, Z. Deng, J. Nie, and J. Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019. - source

In the paper

“In order to assess the severity of this problem, Toutanova and Chen have shown that a simple model based on observable features can easily reach state-of-the-art performance on FB15k.”

What the sources say

• Supported

  The Introduction (§1) says the authors compared a very simple observed-features model with state-of-the-art latent-feature models on FB15K and found that it substantially outperformed them. Section 4.2 likewise reports much higher MRR for the observed-feature models and links their success to direct links in the training graph. (LLM, conf=0.92)

  cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - source

In the paper

“It has been demonstrated that SimplE is fully expressive, and therefore, unlike DistMult, it can model also asymmetric relations.”

What the sources say

• Supported

  Proposition 1 in §5.1 (p. 4) establishes that SimplE can represent any ground truth over entities and relations, supporting full expressivity and thus the ability to model asymmetric relations. The same section says DistMult forces relations to be symmetric; §3 (p. 3) likewise says it can only model symmetric relations. (LLM, conf=0.94)

  cites [27] [27] S. M. Kazemi and D. Poole. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. , pages 4289-4300, 2018. - source

In the paper

“ConvE [11] represents entities and relations as one-dimensional 𝑑 -sized embeddings. When computing the score of a fact, it concatenates and reshapes the head and relation embeddings 𝒉 and 𝒓 into a unique input [ 𝒉 ; 𝒓 ] ; we dub the resulting dimensions 𝑑 𝑚 × 𝑑 𝑛 . This input is let through a convolutional layer with a set 𝝎 of 𝑚 × 𝑛 fi lters, and then through a dense layer with 𝑑 neurons and a”

What the sources say

• Supported

  The model description (p. 3) says ConvE looks up entity and relation embedding vectors, concatenates and reshapes them, applies a 2D convolution, then projects the result into the embedding dimension with a linear layer before scoring it against an object embedding. This supports the attributed architecture. (LLM, conf=0.93)

  cites [11] [11] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018. - source

In the paper

“To resolve that, they build the far more challenging WN18RR dataset by applying a pipeline similar to the one employed for FB15k-237 [57].”

What the sources say

• Supported

  The Knowledge Graph Datasets section (p. 4) says the authors created WN18RR to address flaws in WN18 and make it resistant to completion by a single rule. The Results section (p. 5) explicitly says WN18RR was derived using the same procedure used to derive FB15k-237. (LLM, conf=0.93)

  cites [11] [11] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018. - source
• Partly supported

  Section 4.3 describes a pipeline that removes duplicate and inverse relations to create the more challenging FB15KSelected dataset, with 237 relations. Section 4.2 discusses WN18, but the paper does not say that WN18RR was built using a similar pipeline or assess how challenging WN18RR is relative to WN18. (LLM, conf=0.88)

  cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - sourceExperimental — accuracy not yet measured

In the paper

“HolE [45], instead of using bilinear products, computes circular correlation (denoted by ★ in Table 1) between the embeddings of head and tail entities; then, it performs matrix multiplication with the relation embedding.”

What the sources say

• Supported

  The Holographic Embeddings section (p. 3) says HolE uses circular correlation to represent pairs of entities. The Embedding Models section (p. 4) gives the resulting score as σ(rₚᵀ(eₛ ⋆ eₒ)): the correlation is followed by multiplication by the transposed relation embedding, yielding a scalar. (LLM, conf=0.90)

  cites [45] [45] M. Nickel, L. Rosasco, and T. A. Poggio. Holographic embeddings of knowledge graphs. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. , pages 1955-1961, 2016. - source

In the paper

“FB15k-237 is a subset of FB15k built by Toutanova and Chen [57], inspired by the observation that FB15k suffers from test leakage , consisting in test data being seen by models at training time. In FB15k this issue is due to the presence of relations that are near-identical or the inverse of one another.”

What the sources say

• Supported

  Section 4.2 reports that almost 81% of FB15K test triples have entity pairs directly linked in the training graph, making many cases trivial for a model using those links. Section 4.3 says the authors constructed FB15KSelected by removing near-duplicate and inverse relations and filtering test triples whose entity pairs were directly linked in training. This supports the claimed leakage mechanism. (LLM, conf=0.90)

  cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - source

In the paper

“TuckER [6] relies on the Tucker decomposition [21], which factorizes a tensor into a set of vectors and a smaller shared core 𝑾 . The TuckER model learns 𝑾 jointly with the KG embeddings.”

What the sources say

• Supported

  Section 4 defines TuckER using Tucker decomposition, with entity and relation embeddings and a core tensor W. It describes W as shared across entities and relations and explains that TuckER is trained numerically; together, these passages support that the core is learned jointly with the embeddings. (LLM, conf=0.92)

  cites [6] [6] I. Balazevic, C. Allen, and T. M. Hospedales. Tucker: Tensor factorization for knowledge graph completion. CoRR , abs/1901.09590, 2019. - source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [21] [21] F. L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics , 6(1-4):164-189, 1927. -

In the paper

“Examples of models that have been demonstrated to be fully expressive are SimplE [27], TuckER [6], ComplEx [61] and HolE [60].”

What the sources say

• Supported

  The citation to this paper is attached to SimplE, not to the other models in the list. Section 5.1 (p. 4) explicitly states that Proposition 1 establishes SimplE’s full expressivity and gives the proposition. (LLM, conf=0.94)

  cites [27] [27] S. M. Kazemi and D. Poole. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. , pages 4289-4300, 2018. - source
• Supported

  The citation to this paper is attached to TuckER, not to the other models in the list. Section 5.1, Theorem 1, establishes that a TuckER model can represent any ground truth over a set of entities and relations, which is the paper’s definition of full expressiveness. Page numbers were not provided in our copy. (LLM, conf=0.94)

  cites [6] [6] I. Balazevic, C. Allen, and T. M. Hospedales. Tucker: Tensor factorization for knowledge graph completion. CoRR , abs/1901.09590, 2019. - source
• Supported

  The citation to this paper is attached to ComplEx, not to the other three models. Section 2.2 (p. 4) says its factorization encompasses all possible binary relations, supporting the claim that ComplEx is fully expressive. (LLM, conf=0.88)

  cites [61] [61] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , pages 2071-2080, 2016. - source

In the paper

“we note that ConvKB [41] and CapsE [43] are the only models that require multiple batches for running one prediction, and this may have negatively affected their prediction performance.”

What the sources say

• ?Could not verify

  The survey’s Table 4 (Additional information; page unavailable) lists entity-prediction results for ConvKB and CapsE, but does not describe their batching requirements. A search of our copy for multiple batches and their possible effect on prediction performance found no passage addressing either point. (LLM, conf=0.75)

  cites [41] [41] D. Q. Nguyen, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 327-333, 2018. - sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [43] [43] D. Q. Nguyen, T. Vu, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A capsule network-based embedding model for knowledge graph completion and search personalization. In NAACL-HLT (1) , pages 2180-2189. Association for Computational Linguistics, 2019. -

In the paper

“As a baseline we use AnyBURL [37], a popular LP model based on observable features.”

What the sources say

• Supported

  The Experiments section identifies the cited paper’s approach as AnyBURL (Anytime Bottom-Up Rule Learning). The citation therefore correctly identifies the model that the citing authors say they use as a baseline; the baseline choice is the citing authors’ own. (LLM, conf=0.94)

  cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - source

In the paper

“It has been pointed out that this conversion, dubbed 'Star-to-Clique' (S2C), is irreversible [68].”

What the sources say

• Supported

  The Introduction (p. 1) explicitly says the star-to-clique (S2C) conversion is irreversible. Lemma 2 in §3.2 (p. 5) specifies the qualification: after S2C conversion, the original graph is generally not recoverable. (LLM, conf=0.95)

  cites [68] [68] J. Wen, J. Li, Y. Mao, S. Chen, and R. Zhang. On the representation and embedding of knowledge bases beyond binary relations. In IJCAI , pages 1300-1307. IJCAI/AAAI Press, 2016. - source

In the paper

“We also note that STransE [42], ConvKB [41] and CapsE [43] use transfer learning and require embeddings pre-trained on TransE [8].”

What the sources say

• Partly supported

  Section 4.1 states that the authors initialized STransE’s entity and relation vectors with vectors produced by TransE, supporting the pre-training part for their experiments. The model description presents STransE’s structure without making TransE-pretrained embeddings a requirement, and this paper does not establish the claim about ConvKB or CapsE. (LLM, conf=0.85)

  cites [42] [42] D. Q. Nguyen, K. Sirts, L. Qu, and M. Johnson. Stranse: a novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08140 , 2016. - sourceExperimental — accuracy not yet measured
• Partly supported

  Section 4.2 (page unavailable) says CapsE uses pre-trained word embeddings to initialize entity vectors on WN18RR. It does not say those embeddings were pre-trained on TransE or that they are required. Section 3.1 describes STransE and ConvKB but does not establish transfer learning or a TransE pre-training requirement for either model. (LLM, conf=0.83)

  cites [41] [41] D. Q. Nguyen, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 327-333, 2018. - sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [43] [43] D. Q. Nguyen, T. Vu, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A capsule network-based embedding model for knowledge graph completion and search personalization. In NAACL-HLT (1) , pages 2180-2189. Association for Computational Linguistics, 2019. -

In the paper

“As already mentioned, STransE [42], ConvKB [41] and CapsE [43] require embeddings pre-trained on TransE [8]; we do not include pre-training times in our measurements.”

What the sources say

• Supported

  Section 4.1 states that the authors used entity and relation vectors produced by TransE to initialize STransE. The citation to this paper supports that STransE uses TransE-pre-trained embeddings; the statement about excluding pre-training times describes the citing authors’ own measurements. (LLM, conf=0.91)

  cites [42] [42] D. Q. Nguyen, K. Sirts, L. Qu, and M. Johnson. Stranse: a novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08140 , 2016. - source
• ?Could not verify

  The survey’s Tables 3 and 4 (Additional information; page unavailable) report entity-prediction results, not training-time measurements. A search for pre-training times and their exclusion found no passage addressing either. In the citing sentence, “we do not include pre-training times in our measurements” describes the citing authors’ own measurements, rather than a finding attributed to this survey. (LLM, conf=0.75)

  cites [41] [41] D. Q. Nguyen, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 327-333, 2018. - sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [43] [43] D. Q. Nguyen, T. Vu, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A capsule network-based embedding model for knowledge graph completion and search personalization. In NAACL-HLT (1) , pages 2180-2189. Association for Computational Linguistics, 2019. -

In the paper

“It is also computationally fast: depending on the training setting, in order to learn its rules it can take from a couple of minutes ( 100s setting) to about 3 hours ( 10000s setting).”

What the sources say

• Supported

  Section 5.1 (Anytime Behavior and Runtime) reports rule sets learned after 100 and 10,000 seconds. The Table 2 caption specifies that AnyBURL’s runtimes refer to rule learning only. Those settings correspond to about two minutes and three hours, respectively. (LLM, conf=0.94)

  cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - source

In the paper

“We note that in AnyBURL [37], for datasets FB15k-237 and YAGO3-10 we used training time 1,000 secs whereas the original papers report slightly better results with a training time of 10000s; this is because, due to our already described necessity for full rankings, when using models trained for 10,000 secs prediction times got prohibitively long.”

What the sources say

• Partly supported

  Section 5.1 reports AnyBURL results after both 1,000 and 10,000 seconds of rule learning and says that learning more rules increases application time. It does not say the original paper chose 1,000 seconds instead of 10,000 because prediction became prohibitively slow. In the citing sentence, that choice and its reason are described as the citing authors’ own. (LLM, conf=0.84)

  cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - sourceExperimental — accuracy not yet measured

In the paper

“As a matter of fact, AnyBURL is a very challenging baseline for LP, as it is shown to outperform most latent-featuresbased models.”

What the sources say

• Partly supported

  Section 5.2 says AnyBURL's results are at the same level as, and sometimes slightly better than, most of the embedding models compared. It reports stronger results than ConvE on most datasets and than SimplE, but says ComplEx-N3 performs better than AnyBURL. Thus the paper supports calling AnyBURL a challenging baseline, but 'outperform most' overstates its qualified comparison. (LLM, conf=0.86)

  cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - sourceExperimental — accuracy not yet measured

In the paper

“as noted by Fatemi et al . [15] and Wen et al. [68], 61% of the FreeBase relations are beyond-binary, and the corresponding hyperedges involve more than 1/3rd of the FreeBase entities.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Fatemi et al . [15]
• Partly supported

  The Introduction (p. 1) says that more than one-third of Freebase entities participate in non-binary relations, supporting the entity portion of the claim. Our copy does not report that 61% of Freebase relations are beyond-binary; a search for “61%” found no such statistic. (LLM, conf=0.88)

  cites Wen et al. [68] [68] J. Wen, J. Li, Y. Mao, S. Chen, and R. Zhang. On the representation and embedding of knowledge bases beyond binary relations. In IJCAI , pages 1300-1307. IJCAI/AAAI Press, 2016. - sourceExperimental — accuracy not yet measured

In the paper

“After exploding CVTs into cliques, deduplication is performed: if the same two entities were linked multiple times by the same types of relation using multiple CVTs - e.g. an artist winning multiple awards for the same work - this information is lost, as shown in Figure 4.”

In the paper

“we observe that in AnyBURL predictions, even the full ranking may not contain all entities, as it only includes those supported by at least one rule in the model.”

In the paper

“Not surprisingly, the largest dataset YAGO3-10 usually requires significantly longer training times.”

In the paper

“AnyBURL treats the training time as a configuration parameter, and reaches stateof-the-art performance after just 100s for FB15k and WN18, and 1000s for FB15k237, WN18RR and YAGO3-10.”

In the paper

“We also report that, quite strangely, for a few test facts built with S2C we could not find any CVTs in the FreeBase dump.”

In the paper

“as we have usually observed analogous trends using MRR.”

In the paper

“The generated edges are often redundant, and in the filtering operated to create FB15k-237, many of them are removed.”

In the paper

“In most cases we have not found significant variations; nonetheless, for a few models, min policy yields significantly different results from the other policies.”

In the paper

“In other words, in the new semantics, each fact has happened at least once .”

In the paper

“We observe that ANALOGY behaves particularly well in terms of prediction time”

In the paper

“On YAGO3-10, quite interestingly, ConvKB does not seem to suffer from this issue, while CrossE shows a noticeable difference. CapsE shows the largest problems, with a behaviour akin to the extreme example described above.”

In the paper

“we were able to use RPS as the scoring function of a standalone LP model based on observable features, obtaining acceptable results.”

In the paper

“In FB15k and FB15k-237 both ConvKB and CapsE display huge discrepancies on all metrics, whereas on WN18 and WN18RR the results are almost identical.”

In the paper

“In WN18 and WN18RR the percentage of facts with symmetric relations is quite higher, but no reflexive and transitive relations are found at all.”

In the paper

“In FB15k-237 most of the redundant facts obtained from reified relations have been filtered away, therefore the large majority of test facts belongs to the first bucket.”

In the paper

“In FB15k and WN18, well known for their test leakage, the correlation with performances becomes evidently stronger.”

In the paper

“It is particularly evident in WN18, WN18RR and YAGO3-10, and to a slightly lesser extent in FB15k-237.”

In the paper

“In YAGO3-10 we observe once again TransE [8] and ConvKB [41] having a hard time handling symmetric relations; on these relations, most models actually tend to behave a little worse than their global H@1.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites TransE [8]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites ConvKB [41]

In the paper

“The Geometric family, on the other hand, shows slightly more unstable results.”

In the paper

“In FB15K and FB15K-237 we observe an impressive majority of irreflexive and anti-symmetric relations. Only a few facts involve reflexive, symmetric or transitive relations.”

In the paper

“In YAGO3-10 all test facts feature irreflexive relations; there is a high percentage of facts featuring anti-symmetric relations as well, whereas only a few of them involve symmetric or transitive relations.”

In the paper

“In particular, ComplEx with its N3 regularization displays amazing results on all metrics across all datasets, being the only embedding-based model consistently comparable to the baseline AnyBURL.”

In the paper

“In our experiments we have found that CrossE [72] and, to a much greater extent, ConvKB [41] and CapsE [43], seem sensitive to this issue.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites CrossE [72]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites ConvKB [41]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites CapsE [43]

In the paper

“In YAGO3-10 we still report a significant difference between min and average results, but it is much smaller than before.”

In the paper

“In FB15K all models based on embeddings seem to perform quite well on reflexive relations; on the other hand, the baseline AnyBURL [37] obtains quite bad results on them possibly due to its rule-based approach”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites AnyBURL [37]

In the paper

“In FB15K, in most models we observe that a higher degree generally corresponds to better H@1. The main exceptions are TransE [8], CrossE [72] and STransE [42], that show a stable or even worsening pattern.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites TransE [8]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites CrossE [72]
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites STransE [42]

In the paper

“unfortunately, we found that for our experiments, where the full ranking for all predictions is required for all models in all datasets, PR evaluation is way too time-consuming and thus unfeasible.”

In the paper

“These models show interesting results, but still suffer from some irregularities across metrics and datasets.”

In the paper

“On the contrary, in their FB15k-237, WN18RR and YAGO3-10, which do not suffer from test leakage, models appear to rely also on longer relational paths (3 steps), as well as on the numbers of source/target peers.”

In the paper

“ConvR and RSN display by far the best results in this family, achieving very similar, state-of-the-art performance in FB15k, WN18 and YAGO3-10.”

In the paper

“Deep Learning models, finally, are the most diverse family, with wildly different results depending on the architectural choices of the models and on their implementations.”

In the paper

“We observe that in almost all models and datasets, predictions seem to be facilitated by the presence of source peers and hindered by the presence of target peers.”

In the paper

“At this regard, RotatE shows remarkably consistent performances across all datasets, and it particularly shines when taking into account H@10.”

In the paper

“On the other hand, models such as ConvKB and CapsE often achieve promising results on H@10 and MR metrics, whereas they seem to struggle with H@1 and MRR; furthermore, in some datasets they are clearly hindered by their issues with tie policies described in Section 5.4.1.”

In the paper

“For instance, it has been observed that over 70% of person entities have no known place of birth, and over 99% have no known ethnicity [12, 69] in FreeBase, one of the largest and most widely used KGs for research purposes.”

What the sources say

• ?Not checked — no full text of the sourcecites [12] [12] X. L. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy, T. Strohmann, S. Sun, and W. Zhang. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '14, pages 601-610. ACM, 2014. - source
• ?Not checked — no full text of the sourcecites [69] [69] R. West, E. Gabrilovich, K. Murphy, S. Sun, R. Gupta, and D. Lin. Knowledge base completion via search-based question answering. In Proceedings of the 23rd international conference on World wide web , pages 515-526. ACM, 2014. - source

In the paper

“CrossE [72] is one of the most recent and also most effective models in this group. For each relation it learns an additional relation-specific embedding 𝒄 𝒓 . Given any fact ⟨ ℎ, 𝑟, 𝑡 ⟩ , CrossE uses element-wise products (denoted by ⊙ in Table 1) to combine 𝒉 and 𝒓 with 𝒄 𝒓 . This results in triple-specific embeddings, dubbed interaction embeddings, that are then used in the translation.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [72] [72] W. Zhang, B. Paudel, W. Zhang, A. Bernstein, and H. Chen. Interaction embeddings for prediction and explanation in knowledge graphs. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019 , pages 96-104, 2019.

In the paper

“It has been recently acknowledged by the authors [56] that the test set includes 212 entities that do not appear in the training set, making it impossible to reasonably predict about 6.7% test facts.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [56] [56] TimDettmers. Conve. https://github.com/TimDettmers/ConvE. [Online; accessed 10-October-2019]. -

In the paper

“Training Time and Prediction Time mostly depend ( 𝑖 ) on the model architecture (e.g. deep neural networks may require longer computations due to their inherently longer pipeline of operations);”

In the paper

“Due to the nature of translation, TransE is not able to correctly handle one-to-many and many-to-one relations, as well as symmetric and transitive relations.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [8] [8] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems , pages 2787-2795. NIPS, 2013. -

In the paper

“DistMult [71] forces all relation embeddings to be diagonal matrices, which consistently reduces the space of parameters to be learned, resulting in a much easier model to train.”

What the sources say

• Not checked yetcites [71] [71] B. Yang, W. Yih, X. He, J. Gao, and L. Deng. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. -

In the paper

“Depending on their semantics, relations can be characterized by several properties heavily affecting the ways in which they appear in the facts.”

In the paper

“Therefore, we have modified them in order to extract evaluation results with multiple policies for each model.”

In the paper

“As already pointed out, S2C is not reversible due to its inherent deduplication; therefore, this process often yields multiple CVTs for the same fact.”

In the paper

“( 𝑖𝑖𝑖 ) on the dataset size, namely the number of entities and relations to learn and, for the Training Time, the number of training triples to process.”

In the paper

“As shown by Kadlec et al . [26], LP models tend to be extremely sensitive to hyperparameter tuning, and the hyperparameters for any model often need to be tuned separately for each dataset.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Kadlec et al . [26]

In the paper

“we stress that this depends on the fact that AnyBURL is not designed to generate full rankings, and that using top-k policy lower 𝑘 values would result in much faster computations.”

In the paper

“this may possibly depend on this model being implemented in C++.”

In the paper

“( 𝑖𝑖 ) on the model hyperparameters, such as embedding size and number of negative samples for each positive one;”

In the paper

“For instance, Meilicke et al. [38] have analyzed how their implementations of HolE [45], RESCAL [46] and TransE [8] perform on symmetric relations in various datasets; they report surprisingly bad results for HolE on symmetric relations in FB15K, despite HolE being fully expressive).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [38] [38] C. Meilicke, M. Fink, Y. Wang, D. Ruffinelli, R. Gemulla, and H. Stuckenschmidt. Fine-grained evaluation of rule-and embedding-based systems for knowledge graph completion. In International Semantic Web Conference , pages 3-20. Springer, 2018. -

In the paper

“Such properties have been well known in the LP literature for a long time, because they may lead a relation to form very specific structures and patterns in the graph; this, depending on the model, can make their facts easier or harder to learn and predict.”

In the paper

“While the number of peers only depends on the local neighborhood of the source and target entity, RPS relies on paths that typically have length greater than one.”

In the paper

“For instance, TransE [8] and some of its successors are inherently unable to learn symmetric and transitive relations due to the nature of translation itself.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [8] [8] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems , pages 2787-2795. NIPS, 2013. -

In the paper

“When we have found multiple implementations for the same model, we have always chosen the best best performing one, with the goal of analyzing each model at its best.”

In the paper

“These scores are mainly affected by the embedding dimensions and by the evaluation batch size”

In the paper

“In order to use it as a baseline in our experiments, however, we needed to extract full ranking for each prediction, setting 𝑘 = |E| : this resulted in much longer computation times.”

In the paper

“In order to make our results directly reproducible, we have employed, whenever possible, publicly available implementations.”

In the paper

“for models relying on min policy in their original implementation we report their results obtained with average policy instead”

In the paper

“Therefore, unless differently specified, for models that employed min policy in their original implementation we report their average results instead, as the latter are directly comparable to the results of the other models, whereas the former are not.”

In the paper

“In our trainings, we have relied on the hyperparameter settings reported by the authors for all datasets on which they have run experiments.”

In the paper

“This means that in very few facts, the target entity may not be included even in the full ranking.”

In the paper

“We show this by replicating our experiment using RPS values computed with relational paths of maximum lengths 1 and 2, instead of 3.”

In the paper

“The sigmoid function is not saturating, but for values with very large modulus its slope is almost nil: therefore, due to low-level approximations, it behaves just like a saturating function.”

In the paper

“In the most extreme case, if a model always gives the same score to all entities in the dataset, using min policy it will obtain H@1 = 1.0 (perfect score) whereas using any other policy it would obtain H@1 around 0.0.”

In the paper

“as a consequence, when learning facts with symmetric relations, these models tend to place the possible answers very close to each other in the embedding space. The result would be a crowded area in which the the correct target is often outranked when it comes to H@1, but manages to make it to the top K answers for larger values of K.”

In the paper

“We underline that this decrease in performance is not caused by the target peers just outscoring the target entity: we are taking into account filtered scenario results, therefore target peers, being valid answers to our predictions, do not contribute to the rank computation.”

In the paper

“Saturating activation functions yield the same result for inputs beyond (or below) a certain value.”

In the paper

“In more balanced scenarios, on the contrary, models seem to investigate to a certain extent longer dependencies, as well as to focus more on analogical reasoning supported by examples (such as source peers).”

In the paper

“As already mentioned, source peers work as examples that allow models to characterize more effectively the relation and the target to predict, whereas target peers lead models to confusion, as they try to optimize embeddings to fit too many different answers for the same question.”

In the paper

“The FB15k and FB15k-237 datasets have been built by performing S2C explosion on FreeBase subsamples; this has resulted in greatly altering both the graph structure and its semantics, with overall loss of information.”

In the paper

“Nevertheless, we show that performances computed relying on different policies risk to be not directly comparable to one another, and might not even reflect the actual predictive effectiveness of models.”

In the paper

“approximations may also lead non-saturating functions, such as Sigmoid, behave as saturating in regions where their slope is particularly close to 0.”

In the paper

“In presence of very short patterns providing overwhelming predictive evidence (e.g., the inverse relations that cause test leakage), models seem very prone to just focusing on them, disregarding other forms of reasoning: this can be seen as an unhealthy consequence of the test leakage problem.”

In the paper

“In FB15k-237 and WN18RR, whose filtering processes have cut away the most relevant paths, RSN seems to have a harder time, probably due to its formulation that explicitly leverages paths.”