CoChat
Sign Up
PreprintCited by 34Bayesian Modeling and Causal InferenceMachine Learning and Data ClassificationAdvanced Neural Network Applications

Improving Local Identifiability in Probabilistic Box Embeddings

Open access
Dasgupta, Shib Sankar, Boratko, Michael, Zhang, Dongxu, Vilnis, Luke, Li, Xiang Lorraine, McCallum, Andrew·2020·arXiv
DOI 10.48550/arxiv.2010.04831arXiv 2010.04831
Open this paper in CoChat — free accountOpen PDF

Abstract

Geometric embeddings have recently received attention for their natural ability to represent transitive asymmetric relations via containment. Box embeddings, where objects are represented by n-dimensional hyperrectangles, are a particularly promising example of such an embedding as they are closed under intersection and their volume can be calculated easily, allowing them to naturally represent calibrated probability distributions. The benefits of geometric embeddings also introduce a problem of local identifiability, however, where whole neighborhoods of parameters result in equivalent loss which impedes learning. Prior work addressed some of these issues by using an approximation to Gaussian convolution over the box parameters, however, this intersection operation also increases the sparsity of the gradient. In this work, we model the box parameters with min and max Gumbel distributions, which were chosen such that space is still closed under the operation of the intersection. The calculation of the expected intersection volume involves all parameters, and we demonstrate experimentally that this drastically improves the ability of such models to learn.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.04831

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using Gumbel distribution for boundaries greatly improves box embedding training.

    “This confirms the finding by Dasgupta et al. (2020) that using Gumbel distribution for boundaries greatly improves box embedding training.”

    From Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning (2021) · cites this paper as Dasgupta et al., 2020

  • GumbelBox is a training improvement method for box embeddings.

    “A few training improvement methods for box embeddings have been proposed (Li et al., 2019;Dasgupta et al., 2020), and we make use of the latter, which is termed GumbelBox after the distribution used to model endpoints of boxes.”

    From Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning (2021) · cites this paper as Dasgupta et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 24

  1. [1][1] Ben Athiwaratkun and Andrew Gordon Wilson. Hierarchical density order embeddings. In International Conference on Learning Representations , 2018. -
  2. [3][3] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424 , 2015. - · arXiv
  3. [5][5] Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning , 2018. -
  4. [6][6] Caglar Gulcehre, Marcin Moczulski, Francesco Visin, and Yoshua Bengio. Mollifying networks. arXiv preprint arXiv:1608.04980 , 2016. - · arXiv
  5. [7][7] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997. -
  6. [8][8] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. - · arXiv
  7. [9][9] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. - · arXiv
  8. [10][10] Alice Lai and Julia Hockenmaier. Learning to predict denotational probabilities for modeling entailment. In EACL , 2017. -
  9. [11][11] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017. - · arXiv
  10. [12][12] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research , 18(1):6765-6816, 2017. -
  11. [13][13] Xiang Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. Smoothing the geometry of probabilistic box embeddings. In International Conference on Learning Representations , 2019. -
  12. [14][14] George A Miller. Wordnet: a lexical database for english. Communications of the ACM , 38(11): 39-41, 1995. -
  13. [15][15] Hossein Mobahi. Training recurrent neural networks by diffusion. arXiv preprint arXiv:1601.04114 , 2016. - · arXiv
  14. [16][16] Radford M Neal. Markov chain sampling methods for dirichlet process mixture models. Journal of computational and graphical statistics , 9(2):249-265, 2000. -
  15. [17][17] Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012. -
  16. [18][18] Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Neural Information Processing Systems , 2017. -
  17. [19][19] Dhruvesh Patel, Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, and Andrew McCallum. Representing joint hierarchies with box embeddings. In Automated Knowledge Base Construction , 2020. URL https://openreview.net/forum?id=J246NSqR\_l . -
  18. [20][20] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning , pages 63-71. Springer, 2003. -
  19. [21][21] Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. ICLR , 2020. -
  20. [22][22] Sandeep Subramanian and Soumen Chakrabarti. New embedded representations and evaluation protocols for inferring transitive relations. SIGIR 2018 , 2018. -
  21. [23][23] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning , 2016. -
  22. [24][24] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In International Conference on Learning Representations , 2016. -
  23. [25][25] Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. In ICLR , 2015. -
  24. [26][26] Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic embedding of knowledge graphs with box lattice measures. In Association for Computational Linguistics , 2018.

Cited by

Sorted by citations

Related

Authors
Dasgupta, S. S., Boratko, M., Zhang, D., Vilnis, L., Li, X. L., & McCallum, A.
Affiliations
  • University of Massachusetts Amherst (US)
  • IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
182-192
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.04831
arXiv
2010.04831
OpenAlex
W3098005930
Cited by
33
References
0
Citations by year
2025: 4 · 2024: 11 · 2023: 4 · 2022: 5 · 2021: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
box embeddings, local identifiability, probabilistic embedding, Gumbel distribution, intersection volume, geometric embedding, gradient sparsity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review