Knowledge Graph Embedding for Link Prediction

Probably not a substantial claim: may not pass S3 (load-bearing).

When carefully tuned, DistMult can still reach state-of-the-art performance.

“Despite this flaw, it has been demonstrated by Kadlec et al . [26] that, when carefully tuned, DistMult can still reach state-of-the-art performance.”

• cites [26] [26] R. Kadlec, O. Bajgar, and J. Kleindienst. Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744 , 2017. - sourceSupported

May combine two claims.

ComplEx extends DistMult to complex space, disabling commutativeness and allowing it to model asymmetric relations.

“ComplEx [61], similarly to DistMult, forces each relation embedding to be a diagonal matrix, but extends such formulation in the complex space: 𝒉 ∈ C 𝑑 , 𝒕 ∈ C 𝑑 , 𝒓 ∈ C 𝑑 × 𝑑 . In the complex space, the bilinear product becomes a Hermitian product, where in lieu of the traditional 𝒕 , its conjugate-transpose 𝒕 is employed. This disables the commutativeness above mentioned for the scoring function”

• cites [61] [61] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , pages 2071-2080, 2016. - sourceSupported

May combine two claims.

STransE associates to each relation two additional matrices to pre-multiply head and tail, alleviating TransE issues on 1-to-many, many-to-one and many-to-many relations.

“STransE [42], in addition to the 𝑑 -sized embeddings seen in TransE, associates to each relation 𝑟 two additional 𝑑 × 𝑑 independent matrices 𝑾 𝒉 𝒓 and 𝑾 𝒕 𝒓 . When computing the score of a fact ⟨ ℎ, 𝑟, 𝑡 ⟩ , before operating the usual translation, 𝒉 is pre-multiplied by 𝑾 𝒉 𝒓 and 𝒕 by 𝑾 𝒕 𝒓 ; this amounts to use relation-specific embeddings for the head and tail, alleviating the issues suffered by”

• cites [42] [42] D. Q. Nguyen, K. Sirts, L. Qu, and M. Johnson. Stranse: a novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08140 , 2016. - sourceSupported

Classified as probably not a substantial claim.

A simple rule-based model based on inverse relation detection, called Inverse Model, achieves state-of-the-art results in both WN18 and FB15k.

“They demonstrate the severity of said leakage by showing that a simple rule-based model based on inverse relation detection, dubbed Inverse Model, achieves state-of-the-art results in both WN18 and FB15k.”

• cites [11] [11] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018. - sourceSupported

May combine two claims.

FB15k-237 was built to be a more challenging dataset by selecting facts from FB15k involving the 401 largest relations and removing all equivalent or inverse relations.

“FB15k-237 was built to be a more challenging dataset: the authors first selected facts from FB15k involving the 401 largest relations and removed all equivalent or inverse relations.”

• cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - sourceSupported

May combine two claims.

RotatE represents relations as rotations in a complex latent space and can model correctly numerous relational patterns, such as symmetry/anti-symmetry, inversion and composition.

“RotatE [55] represents relations as rotations in a complex latent space, with 𝒉 , 𝒓 and 𝒕 all belonging to C 𝑑 . The 𝒓 embedding is a rotation vector: in all its elements, the complex component conveys the rotation along that axis, whereas the real component is always equal to 1. The rotation 𝒓 is applied to 𝒉 by operating an element-wise product (once again noted with ⊙ in 1). L1 norm is used for”

• cites [55] [55] Z. Sun, Z. Deng, J. Nie, and J. Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019. - sourceSupported

Classified as probably not a substantial claim.

A simple model based on observable features can easily reach state-of-the-art performance on FB15k.

“In order to assess the severity of this problem, Toutanova and Chen have shown that a simple model based on observable features can easily reach state-of-the-art performance on FB15k.”

• cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - sourceSupported

May combine two claims.

SimplE is fully expressive and can model asymmetric relations.

“It has been demonstrated that SimplE is fully expressive, and therefore, unlike DistMult, it can model also asymmetric relations.”

• cites [27] [27] S. M. Kazemi and D. Poole. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. , pages 4289-4300, 2018. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ConvE represents entities and relations as one-dimensional embeddings and uses a convolutional layer followed by a dense layer to compute fact scores.

“ConvE [11] represents entities and relations as one-dimensional 𝑑 -sized embeddings. When computing the score of a fact, it concatenates and reshapes the head and relation embeddings 𝒉 and 𝒓 into a unique input [ 𝒉 ; 𝒓 ] ; we dub the resulting dimensions 𝑑 𝑚 × 𝑑 𝑛 . This input is let through a convolutional layer with a set 𝝎 of 𝑚 × 𝑛 fi lters, and then through a dense layer with 𝑑 neurons and a”

• cites [11] [11] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

WN18RR was built to be a more challenging dataset than WN18 by applying a pipeline similar to the one employed for FB15k-237.

“To resolve that, they build the far more challenging WN18RR dataset by applying a pipeline similar to the one employed for FB15k-237 [57].”

• cites [11] [11] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence , 2018. - sourceSupported
• cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - sourcePartly supported

  Section 4.3 describes a pipeline that removes duplicate and inverse relations to create the more challenging FB15KSelected dataset, with 237 relations. Section 4.2 discusses WN18, but the paper does not say that WN18RR was built using a similar pipeline or assess how challenging WN18RR is relative to WN18. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

HolE computes circular correlation between head and tail embeddings and then matrix multiplication with the relation embedding.

“HolE [45], instead of using bilinear products, computes circular correlation (denoted by ★ in Table 1) between the embeddings of head and tail entities; then, it performs matrix multiplication with the relation embedding.”

• cites [45] [45] M. Nickel, L. Rosasco, and T. A. Poggio. Holographic embeddings of knowledge graphs. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. , pages 1955-1961, 2016. - sourceSupported

May combine two claims.

FB15k suffers from test leakage, consisting in test data being seen by models at training time, due to relations that are near-identical or the inverse of one another.

“FB15k-237 is a subset of FB15k built by Toutanova and Chen [57], inspired by the observation that FB15k suffers from test leakage , consisting in test data being seen by models at training time. In FB15k this issue is due to the presence of relations that are near-identical or the inverse of one another.”

• cites [57] [57] K. Toutanova and D. Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality , pages 57-66, 2015. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

TuckER relies on Tucker decomposition and learns a shared core tensor jointly with KG embeddings.

“TuckER [6] relies on the Tucker decomposition [21], which factorizes a tensor into a set of vectors and a smaller shared core 𝑾 . The TuckER model learns 𝑾 jointly with the KG embeddings.”

• cites [6] [6] I. Balazevic, C. Allen, and T. M. Hospedales. Tucker: Tensor factorization for knowledge graph completion. CoRR , abs/1901.09590, 2019. - sourceSupported
• cites [21] [21] F. L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics , 6(1-4):164-189, 1927. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

SimplE, TuckER, ComplEx and HolE have been demonstrated to be fully expressive.

“Examples of models that have been demonstrated to be fully expressive are SimplE [27], TuckER [6], ComplEx [61] and HolE [60].”

• cites [27] [27] S. M. Kazemi and D. Poole. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. , pages 4289-4300, 2018. - sourceSupported
• cites [6] [6] I. Balazevic, C. Allen, and T. M. Hospedales. Tucker: Tensor factorization for knowledge graph completion. CoRR , abs/1901.09590, 2019. - sourceSupported
• cites [61] [61] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , pages 2071-2080, 2016. - sourceSupported

May combine two claims.

ConvKB and CapsE are the only models that require multiple batches for running one prediction, and this may have negatively affected their prediction performance.

“we note that ConvKB [41] and CapsE [43] are the only models that require multiple batches for running one prediction, and this may have negatively affected their prediction performance.”

• cites [41] [41] D. Q. Nguyen, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 327-333, 2018. - sourceCould not verify

  The survey’s Table 4 (Additional information; page unavailable) lists entity-prediction results for ConvKB and CapsE, but does not describe their batching requirements. A search of our copy for multiple batches and their possible effect on prediction performance found no passage addressing either point. (LLM, conf=0.75)

  Experimental — accuracy not yet measured
• cites [43] [43] D. Q. Nguyen, T. Vu, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A capsule network-based embedding model for knowledge graph completion and search personalization. In NAACL-HLT (1) , pages 2180-2189. Association for Computational Linguistics, 2019. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper uses AnyBURL as a baseline.

“As a baseline we use AnyBURL [37], a popular LP model based on observable features.”

• cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - sourceSupported

Classified as probably not a substantial claim.

The conversion dubbed Star-to-Clique (S2C) is irreversible.

“It has been pointed out that this conversion, dubbed 'Star-to-Clique' (S2C), is irreversible [68].”

• cites [68] [68] J. Wen, J. Li, Y. Mao, S. Chen, and R. Zhang. On the representation and embedding of knowledge bases beyond binary relations. In IJCAI , pages 1300-1307. IJCAI/AAAI Press, 2016. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

STransE, ConvKB and CapsE use transfer learning and require embeddings pre-trained on TransE.

“We also note that STransE [42], ConvKB [41] and CapsE [43] use transfer learning and require embeddings pre-trained on TransE [8].”

• cites [42] [42] D. Q. Nguyen, K. Sirts, L. Qu, and M. Johnson. Stranse: a novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08140 , 2016. - sourcePartly supported

  Section 4.1 states that the authors initialized STransE’s entity and relation vectors with vectors produced by TransE, supporting the pre-training part for their experiments. The model description presents STransE’s structure without making TransE-pretrained embeddings a requirement, and this paper does not establish the claim about ConvKB or CapsE. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [41] [41] D. Q. Nguyen, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 327-333, 2018. - sourcePartly supported

  Section 4.2 (page unavailable) says CapsE uses pre-trained word embeddings to initialize entity vectors on WN18RR. It does not say those embeddings were pre-trained on TransE or that they are required. Section 3.1 describes STransE and ConvKB but does not establish transfer learning or a TransE pre-training requirement for either model. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [43] [43] D. Q. Nguyen, T. Vu, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A capsule network-based embedding model for knowledge graph completion and search personalization. In NAACL-HLT (1) , pages 2180-2189. Association for Computational Linguistics, 2019. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper does not include pre-training times in its measurements for STransE, ConvKB and CapsE.

“As already mentioned, STransE [42], ConvKB [41] and CapsE [43] require embeddings pre-trained on TransE [8]; we do not include pre-training times in our measurements.”

• cites [42] [42] D. Q. Nguyen, K. Sirts, L. Qu, and M. Johnson. Stranse: a novel embedding model of entities and relationships in knowledge bases. arXiv preprint arXiv:1606.08140 , 2016. - sourceSupported
• cites [41] [41] D. Q. Nguyen, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 327-333, 2018. - sourceCould not verify

  The survey’s Tables 3 and 4 (Additional information; page unavailable) report entity-prediction results, not training-time measurements. A search for pre-training times and their exclusion found no passage addressing either. In the citing sentence, “we do not include pre-training times in our measurements” describes the citing authors’ own measurements, rather than a finding attributed to this survey. (LLM, conf=0.75)

  Experimental — accuracy not yet measured
• cites [43] [43] D. Q. Nguyen, T. Vu, T. D. Nguyen, D. Q. Nguyen, and D. Q. Phung. A capsule network-based embedding model for knowledge graph completion and search personalization. In NAACL-HLT (1) , pages 2180-2189. Association for Computational Linguistics, 2019. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

AnyBURL is computationally fast: depending on the training setting, it can take from a couple of minutes (100s setting) to about 3 hours (10000s setting) to learn its rules.

“It is also computationally fast: depending on the training setting, in order to learn its rules it can take from a couple of minutes ( 100s setting) to about 3 hours ( 10000s setting).”

• cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

For AnyBURL on datasets FB15k-237 and YAGO3-10, the paper used training time 1,000 secs instead of 10,000 secs because 10,000 secs made prediction times prohibitively long.

“We note that in AnyBURL [37], for datasets FB15k-237 and YAGO3-10 we used training time 1,000 secs whereas the original papers report slightly better results with a training time of 10000s; this is because, due to our already described necessity for full rankings, when using models trained for 10,000 secs prediction times got prohibitively long.”

• cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - sourcePartly supported

  Section 5.1 reports AnyBURL results after both 1,000 and 10,000 seconds of rule learning and says that learning more rules increases application time. It does not say the original paper chose 1,000 seconds instead of 10,000 because prediction became prohibitively slow. In the citing sentence, that choice and its reason are described as the citing authors’ own. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

AnyBURL is a very challenging baseline for LP, as it is shown to outperform most latent-features-based models.

“As a matter of fact, AnyBURL is a very challenging baseline for LP, as it is shown to outperform most latent-featuresbased models.”

• cites [37] [37] C. Meilicke, M. W. Chekol, D. Ruffinelli, and H. Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. 2019. - sourcePartly supported

  Section 5.2 says AnyBURL's results are at the same level as, and sometimes slightly better than, most of the embedding models compared. It reports stronger results than ConvE on most datasets and than SimplE, but says ComplEx-N3 performs better than AnyBURL. Thus the paper supports calling AnyBURL a challenging baseline, but 'outperform most' overstates its qualified comparison. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

61% of FreeBase relations are beyond-binary, and corresponding hyperedges involve more than 1/3rd of FreeBase entities.

“as noted by Fatemi et al . [15] and Wen et al. [68], 61% of the FreeBase relations are beyond-binary, and the corresponding hyperedges involve more than 1/3rd of the FreeBase entities.”

• cites Fatemi et al . [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wen et al. [68] [68] J. Wen, J. Li, Y. Mao, S. Chen, and R. Zhang. On the representation and embedding of knowledge bases beyond binary relations. In IJCAI , pages 1300-1307. IJCAI/AAAI Press, 2016. - sourcePartly supported

  The Introduction (p. 1) says that more than one-third of Freebase entities participate in non-binary relations, supporting the entity portion of the claim. Our copy does not report that 61% of Freebase relations are beyond-binary; a search for “61%” found no such statistic. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

After exploding CVTs into cliques, deduplication is performed, and if the same two entities were linked multiple times by the same types of relation using multiple CVTs, this information is lost.

“After exploding CVTs into cliques, deduplication is performed: if the same two entities were linked multiple times by the same types of relation using multiple CVTs - e.g. an artist winning multiple awards for the same work - this information is lost, as shown in Figure 4.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In AnyBURL predictions, even the full ranking may not contain all entities, as it only includes those supported by at least one rule in the model.

“we observe that in AnyBURL predictions, even the full ranking may not contain all entities, as it only includes those supported by at least one rule in the model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The largest dataset YAGO3-10 usually requires significantly longer training times.

“Not surprisingly, the largest dataset YAGO3-10 usually requires significantly longer training times.”

May combine two claims.

AnyBURL reaches state-of-the-art performance after just 100s for FB15k and WN18, and 1000s for FB15k237, WN18RR and YAGO3-10.

“AnyBURL treats the training time as a configuration parameter, and reaches stateof-the-art performance after just 100s for FB15k and WN18, and 1000s for FB15k237, WN18RR and YAGO3-10.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For a few test facts built with S2C, no CVTs could be found in the FreeBase dump.

“We also report that, quite strangely, for a few test facts built with S2C we could not find any CVTs in the FreeBase dump.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper has usually observed analogous trends using MRR as with H@1.

“as we have usually observed analogous trends using MRR.”

May combine two claims.

The edges generated by S2C are often redundant, and in the filtering operated to create FB15k-237, many of them are removed.

“The generated edges are often redundant, and in the filtering operated to create FB15k-237, many of them are removed.”

May combine two claims.

In most cases there were no significant variations across tie policies; for a few models, min policy yields significantly different results from the other policies.

“In most cases we have not found significant variations; nonetheless, for a few models, min policy yields significantly different results from the other policies.”

Classified as probably not a substantial claim.

In the new semantics after S2C, each fact has happened at least once.

“In other words, in the new semantics, each fact has happened at least once .”

Probably not a substantial claim: may not pass S3 (load-bearing).

ANALOGY behaves particularly well in terms of prediction time.

“We observe that ANALOGY behaves particularly well in terms of prediction time”

May combine two claims.

On YAGO3-10, ConvKB does not seem to suffer from the tie policy issue, while CrossE shows a noticeable difference, and CapsE shows the largest problems.

“On YAGO3-10, quite interestingly, ConvKB does not seem to suffer from this issue, while CrossE shows a noticeable difference. CapsE shows the largest problems, with a behaviour akin to the extreme example described above.”

The quote only partly states this claim.

A standalone LP model based on observable features using RPS as scoring function with relational paths up to 2 steps obtained acceptable results on FB15k and WN18 (Table 4).

“we were able to use RPS as the scoring function of a standalone LP model based on observable features, obtaining acceptable results.”

May combine two claims.

In FB15k and FB15k-237, ConvKB and CapsE display huge discrepancies between tie policies on all metrics, whereas on WN18 and WN18RR results are almost identical.

“In FB15k and FB15k-237 both ConvKB and CapsE display huge discrepancies on all metrics, whereas on WN18 and WN18RR the results are almost identical.”

May combine two claims.

In WN18 and WN18RR, the percentage of facts with symmetric relations is quite higher than in FB15k/FB15k-237, but no reflexive and transitive relations are found at all.

“In WN18 and WN18RR the percentage of facts with symmetric relations is quite higher, but no reflexive and transitive relations are found at all.”

May combine two claims.

In FB15k-237, most redundant facts obtained from reified relations have been filtered away, so the large majority of test facts belongs to the first bucket (degree 1).

“In FB15k-237 most of the redundant facts obtained from reified relations have been filtered away, therefore the large majority of test facts belongs to the first bucket.”

The quote only partly states this claim.

In FB15k and WN18, well known for their test leakage, the correlation with performances becomes evidently stronger when using RPS with max lengths 1 and 2.

“In FB15k and WN18, well known for their test leakage, the correlation with performances becomes evidently stronger.”

May combine two claims.

The correlation is particularly evident in WN18, WN18RR and YAGO3-10, and to a slightly lesser extent in FB15k-237.

“It is particularly evident in WN18, WN18RR and YAGO3-10, and to a slightly lesser extent in FB15k-237.”

May combine two claims.

In YAGO3-10, TransE and ConvKB have a hard time handling symmetric relations; on these relations, most models tend to behave a little worse than their global H@1.

“In YAGO3-10 we observe once again TransE [8] and ConvKB [41] having a hard time handling symmetric relations; on these relations, most models actually tend to behave a little worse than their global H@1.”

• cites TransE [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ConvKB [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The quote only partly states this claim.

The Geometric family shows slightly more unstable results than Tensor Decomposition.

“The Geometric family, on the other hand, shows slightly more unstable results.”

May combine two claims.

In FB15k and FB15k-237, an impressive majority of relations are irreflexive and anti-symmetric; only a few facts involve reflexive, symmetric, or transitive relations.

“In FB15K and FB15K-237 we observe an impressive majority of irreflexive and anti-symmetric relations. Only a few facts involve reflexive, symmetric or transitive relations.”

May combine two claims.

In YAGO3-10, all test facts feature irreflexive relations; there is a high percentage of facts featuring anti-symmetric relations, whereas only a few involve symmetric or transitive relations.

“In YAGO3-10 all test facts feature irreflexive relations; there is a high percentage of facts featuring anti-symmetric relations as well, whereas only a few of them involve symmetric or transitive relations.”

May combine two claims.

ComplEx with N3 regularization displays amazing results on all metrics across all datasets, being the only embedding-based model consistently comparable to AnyBURL.

“In particular, ComplEx with its N3 regularization displays amazing results on all metrics across all datasets, being the only embedding-based model consistently comparable to the baseline AnyBURL.”

May combine two claims.

In our experiments, CrossE, ConvKB, and CapsE seem sensitive to the tie policy issue, with ConvKB and CapsE more so.

“In our experiments we have found that CrossE [72] and, to a much greater extent, ConvKB [41] and CapsE [43], seem sensitive to this issue.”

• cites CrossE [72]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ConvKB [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites CapsE [43]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In YAGO3-10, for CapsE 'Leaky', there is still a significant difference between min and average, but much smaller than before.

“In YAGO3-10 we still report a significant difference between min and average results, but it is much smaller than before.”

May combine two claims.

In FB15k, all embedding-based models perform quite well on reflexive relations, whereas AnyBURL obtains quite bad results on them.

“In FB15K all models based on embeddings seem to perform quite well on reflexive relations; on the other hand, the baseline AnyBURL [37] obtains quite bad results on them possibly due to its rule-based approach”

• cites AnyBURL [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In FB15k, for most models, a higher degree of the original reified relation generally corresponds to better H@1; exceptions are TransE, CrossE, and STransE, which show a stable or worsening pattern.

“In FB15K, in most models we observe that a higher degree generally corresponds to better H@1. The main exceptions are TransE [8], CrossE [72] and STransE [42], that show a stable or even worsening pattern.”

• cites TransE [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites CrossE [72]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites STransE [42]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

PR evaluation is too time-consuming and unfeasible for our experiments.

“unfortunately, we found that for our experiments, where the full ranking for all predictions is required for all models in all datasets, PR evaluation is way too time-consuming and thus unfeasible.”

May combine two claims.

Translational models show interesting results but suffer from irregularities across metrics and datasets.

“These models show interesting results, but still suffer from some irregularities across metrics and datasets.”

May combine two claims.

In FB15k-237, WN18RR, and YAGO3-10, which do not suffer from test leakage, models appear to rely also on longer relational paths (3 steps) and on numbers of source/target peers.

“On the contrary, in their FB15k-237, WN18RR and YAGO3-10, which do not suffer from test leakage, models appear to rely also on longer relational paths (3 steps), as well as on the numbers of source/target peers.”

May combine two claims.

ConvR and RSN display by far the best results in the Deep Learning family, achieving very similar, state-of-the-art performance in FB15k, WN18, and YAGO3-10.

“ConvR and RSN display by far the best results in this family, achieving very similar, state-of-the-art performance in FB15k, WN18 and YAGO3-10.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Deep Learning models are the most diverse family, with wildly different results depending on architectural choices and implementations.

“Deep Learning models, finally, are the most diverse family, with wildly different results depending on the architectural choices of the models and on their implementations.”

May combine two claims.

In almost all models and datasets, predictions are facilitated by the presence of source peers and hindered by the presence of target peers.

“We observe that in almost all models and datasets, predictions seem to be facilitated by the presence of source peers and hindered by the presence of target peers.”

May combine two claims.

RotatE shows remarkably consistent performances across all datasets, particularly on H@10.

“At this regard, RotatE shows remarkably consistent performances across all datasets, and it particularly shines when taking into account H@10.”

May combine two claims.

ConvKB and CapsE often achieve promising results on H@10 and MR metrics, but struggle with H@1 and MRR; in some datasets they are hindered by tie policy issues.

“On the other hand, models such as ConvKB and CapsE often achieve promising results on H@10 and MR metrics, whereas they seem to struggle with H@1 and MRR; furthermore, in some datasets they are clearly hindered by their issues with tie policies described in Section 5.4.1.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In FreeBase, over 70% of person entities have no known place of birth, and over 99% have no known ethnicity.

“For instance, it has been observed that over 70% of person entities have no known place of birth, and over 99% have no known ethnicity [12, 69] in FreeBase, one of the largest and most widely used KGs for research purposes.”

• cites [12] [12] X. L. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy, T. Strohmann, S. Sun, and W. Zhang. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '14, pages 601-610. ACM, 2014. - sourceNot checked — no full text of the source
• cites [69] [69] R. West, E. Gabrilovich, K. Murphy, S. Sun, R. Gupta, and D. Lin. Knowledge base completion via search-based question answering. In Proceedings of the 23rd international conference on World wide web , pages 515-526. ACM, 2014. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CrossE learns an additional relation-specific embedding and uses element-wise products to combine head and relation, resulting in triple-specific interaction embeddings.

“CrossE [72] is one of the most recent and also most effective models in this group. For each relation it learns an additional relation-specific embedding 𝒄 𝒓 . Given any fact ⟨ ℎ, 𝑟, 𝑡 ⟩ , CrossE uses element-wise products (denoted by ⊙ in Table 1) to combine 𝒉 and 𝒓 with 𝒄 𝒓 . This results in triple-specific embeddings, dubbed interaction embeddings, that are then used in the translation.”

• cites [72] [72] W. Zhang, B. Paudel, W. Zhang, A. Bernstein, and H. Chen. Interaction embeddings for prediction and explanation in knowledge graphs. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019 , pages 96-104, 2019.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

WN18RR test set includes 212 entities that do not appear in the training set, making it impossible to reasonably predict about 6.7% of test facts.

“It has been recently acknowledged by the authors [56] that the test set includes 212 entities that do not appear in the training set, making it impossible to reasonably predict about 6.7% test facts.”

• cites [56] [56] TimDettmers. Conve. https://github.com/TimDettmers/ConvE. [Online; accessed 10-October-2019]. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Training Time and Prediction Time mostly depend on the model architecture.

“Training Time and Prediction Time mostly depend ( 𝑖 ) on the model architecture (e.g. deep neural networks may require longer computations due to their inherently longer pipeline of operations);”

May combine two claims.

TransE is not able to correctly handle one-to-many and many-to-one relations, as well as symmetric and transitive relations.

“Due to the nature of translation, TransE is not able to correctly handle one-to-many and many-to-one relations, as well as symmetric and transitive relations.”

• cites [8] [8] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems , pages 2787-2795. NIPS, 2013. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

DistMult forces all relation embeddings to be diagonal matrices, reducing the parameter space and making the model easier to train.

“DistMult [71] forces all relation embeddings to be diagonal matrices, which consistently reduces the space of parameters to be learned, resulting in a much easier model to train.”

• cites [71] [71] B. Yang, W. Yih, X. He, J. Gao, and L. Deng. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Depending on their semantics, relations can be characterized by several properties heavily affecting the ways in which they appear in the facts.

“Depending on their semantics, relations can be characterized by several properties heavily affecting the ways in which they appear in the facts.”

Classified as probably not a substantial claim.

The paper modified implementations to extract evaluation results with multiple policies for each model.

“Therefore, we have modified them in order to extract evaluation results with multiple policies for each model.”

May combine two claims.

Because S2C is not reversible due to its inherent deduplication, the process of recovering CVTs often yields multiple CVTs for the same fact.

“As already pointed out, S2C is not reversible due to its inherent deduplication; therefore, this process often yields multiple CVTs for the same fact.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Training Time and Prediction Time mostly depend on the dataset size, namely the number of entities and relations and, for Training Time, the number of training triples.

“( 𝑖𝑖𝑖 ) on the dataset size, namely the number of entities and relations to learn and, for the Training Time, the number of training triples to process.”

May combine two claims.

LP models tend to be extremely sensitive to hyperparameter tuning, and hyperparameters often need to be tuned separately for each dataset.

“As shown by Kadlec et al . [26], LP models tend to be extremely sensitive to hyperparameter tuning, and the hyperparameters for any model often need to be tuned separately for each dataset.”

• cites Kadlec et al . [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

AnyBURL's higher prediction times depend on the fact that AnyBURL is not designed to generate full rankings, and using top-k policy with lower k values would result in much faster computations.

“we stress that this depends on the fact that AnyBURL is not designed to generate full rankings, and that using top-k policy lower 𝑘 values would result in much faster computations.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

ANALOGY's good prediction time may possibly depend on this model being implemented in C++.

“this may possibly depend on this model being implemented in C++.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Training Time and Prediction Time mostly depend on the model hyperparameters, such as embedding size and number of negative samples per positive.

“( 𝑖𝑖 ) on the model hyperparameters, such as embedding size and number of negative samples for each positive one;”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Meilicke et al. analyzed how their implementations of HolE, RESCAL and TransE perform on symmetric relations in various datasets and reported surprisingly bad results for HolE on symmetric relations in FB15K, despite HolE being fully expressive.

“For instance, Meilicke et al. [38] have analyzed how their implementations of HolE [45], RESCAL [46] and TransE [8] perform on symmetric relations in various datasets; they report surprisingly bad results for HolE on symmetric relations in FB15K, despite HolE being fully expressive).”

• cites [38] [38] C. Meilicke, M. Fink, Y. Wang, D. Ruffinelli, R. Gemulla, and H. Stuckenschmidt. Fine-grained evaluation of rule-and embedding-based systems for knowledge graph completion. In International Semantic Web Conference , pages 3-20. Springer, 2018. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Relation properties may lead a relation to form very specific structures and patterns in the graph, which, depending on the model, can make their facts easier or harder to learn and predict.

“Such properties have been well known in the LP literature for a long time, because they may lead a relation to form very specific structures and patterns in the graph; this, depending on the model, can make their facts easier or harder to learn and predict.”

May combine two claims.

The number of peers only depends on the local neighborhood of the source and target entity, whereas RPS relies on paths that typically have length greater than one.

“While the number of peers only depends on the local neighborhood of the source and target entity, RPS relies on paths that typically have length greater than one.”

Probably not a substantial claim: may not pass S3 (load-bearing).

TransE and some of its successors are inherently unable to learn symmetric and transitive relations due to the nature of translation itself.

“For instance, TransE [8] and some of its successors are inherently unable to learn symmetric and transitive relations due to the nature of translation itself.”

• cites [8] [8] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems , pages 2787-2795. NIPS, 2013. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

When multiple implementations for the same model were found, the paper always chose the best performing one to analyze each model at its best.

“When we have found multiple implementations for the same model, we have always chosen the best best performing one, with the goal of analyzing each model at its best.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Prediction time scores are mainly affected by the embedding dimensions and by the evaluation batch size.

“These scores are mainly affected by the embedding dimensions and by the evaluation batch size”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In order to use AnyBURL as a baseline, the paper needed to extract full ranking for each prediction, setting k = |E|, which resulted in much longer computation times.

“In order to use it as a baseline in our experiments, however, we needed to extract full ranking for each prediction, setting 𝑘 = |E| : this resulted in much longer computation times.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper employed publicly available implementations whenever possible to make its results directly reproducible.

“In order to make our results directly reproducible, we have employed, whenever possible, publicly available implementations.”

Classified as probably not a substantial claim.

For models relying on min policy in their original implementation, the paper reports results obtained with average policy instead.

“for models relying on min policy in their original implementation we report their results obtained with average policy instead”

Classified as probably not a substantial claim. May combine two claims.

For models that employed min policy in their original implementation, the paper reports their average results instead, because average results are directly comparable to the results of other models whereas min policy results are not.

“Therefore, unless differently specified, for models that employed min policy in their original implementation we report their average results instead, as the latter are directly comparable to the results of the other models, whereas the former are not.”

Classified as probably not a substantial claim.

In its trainings, the paper relied on the hyperparameter settings reported by the authors for all datasets on which they had run experiments.

“In our trainings, we have relied on the hyperparameter settings reported by the authors for all datasets on which they have run experiments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In very few facts, the target entity may not be included even in the full ranking.

“This means that in very few facts, the target entity may not be included even in the full ranking.”

Classified as probably not a substantial claim.

The paper replicated the experiment using RPS values computed with relational paths of maximum lengths 1 and 2, instead of 3.

“We show this by replicating our experiment using RPS values computed with relational paths of maximum lengths 1 and 2, instead of 3.”

May combine two claims.

The sigmoid function is not saturating, but for values with very large modulus its slope is almost nil, so due to low-level approximations it behaves like a saturating function.

“The sigmoid function is not saturating, but for values with very large modulus its slope is almost nil: therefore, due to low-level approximations, it behaves just like a saturating function.”

May combine two claims.

In the extreme case where a model always gives the same score to all entities, min policy yields H@1 = 1.0 while any other policy yields H@1 around 0.0.

“In the most extreme case, if a model always gives the same score to all entities in the dataset, using min policy it will obtain H@1 = 1.0 (perfect score) whereas using any other policy it would obtain H@1 around 0.0.”

May combine two claims.

As a consequence of the S2C transformation generating many symmetric relations and of TransE, STransE, and CrossE representing symmetric relations with very small vectors, when learning facts with symmetric relations these models tend to place possible answers very close to each other in the embedding space, causing a crowded area where the correct target is often outranked in H@1 but manages to m

“as a consequence, when learning facts with symmetric relations, these models tend to place the possible answers very close to each other in the embedding space. The result would be a crowded area in which the the correct target is often outranked when it comes to H@1, but manages to make it to the top K answers for larger values of K.”

• AssumptionRests on an assumption: TransE, STransE, and CrossE are naturally inclined to represent symmetric relations with very small vectors in the embedding space.
• AssumptionRests on an assumption: Due to the S2C transformation, original reified relations tend to generate a high number of facts containing symmetric relations.

May combine two claims.

The decrease in performance with more target peers is not caused by target peers outscoring the target entity, because filtered scenario results exclude target peers from rank computation.

“We underline that this decrease in performance is not caused by the target peers just outscoring the target entity: we are taking into account filtered scenario results, therefore target peers, being valid answers to our predictions, do not contribute to the rank computation.”

Classified as probably not a substantial claim.

Saturating activation functions yield the same result for inputs beyond (or below) a certain value.

“Saturating activation functions yield the same result for inputs beyond (or below) a certain value.”

May combine two claims.

In more balanced scenarios, models seem to investigate longer dependencies and focus more on analogical reasoning supported by examples.

“In more balanced scenarios, on the contrary, models seem to investigate to a certain extent longer dependencies, as well as to focus more on analogical reasoning supported by examples (such as source peers).”

May combine two claims.

Source peers work as examples that allow models to characterize more effectively the relation and target to predict, whereas target peers lead models to confusion because they try to optimize embeddings to fit too many different answers for the same question.

“As already mentioned, source peers work as examples that allow models to characterize more effectively the relation and the target to predict, whereas target peers lead models to confusion, as they try to optimize embeddings to fit too many different answers for the same question.”

May combine two claims.

The S2C explosion used to build FB15k and FB15k-237 has greatly altered both graph structure and semantics, with overall loss of information.

“The FB15k and FB15k-237 datasets have been built by performing S2C explosion on FreeBase subsamples; this has resulted in greatly altering both the graph structure and its semantics, with overall loss of information.”

May combine two claims.

Performances computed relying on different policies risk being not directly comparable and might not reflect actual predictive effectiveness.

“Nevertheless, we show that performances computed relying on different policies risk to be not directly comparable to one another, and might not even reflect the actual predictive effectiveness of models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Approximations may lead non-saturating functions such as Sigmoid to behave as saturating in regions where their slope is close to 0.

“approximations may also lead non-saturating functions, such as Sigmoid, behave as saturating in regions where their slope is particularly close to 0.”

May combine two claims.

In presence of very short patterns providing overwhelming predictive evidence, models seem prone to focusing on them and disregarding other forms of reasoning, which can be seen as an unhealthy consequence of test leakage.

“In presence of very short patterns providing overwhelming predictive evidence (e.g., the inverse relations that cause test leakage), models seem very prone to just focusing on them, disregarding other forms of reasoning: this can be seen as an unhealthy consequence of the test leakage problem.”

May combine two claims.

In FB15k-237 and WN18RR, RSN seems to have a harder time, probably due to its formulation that explicitly leverages paths.

“In FB15k-237 and WN18RR, whose filtering processes have cut away the most relevant paths, RSN seems to have a harder time, probably due to its formulation that explicitly leverages paths.”