Order-Embeddings of Images and Language
Open accessAbstract
Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images. In this paper we advocate for explicitly modeling the partial order structure of this hierarchy. Towards this goal, we introduce a general method for learning ordered representations, and show how it can be applied to a variety of tasks involving images and language. We show that the resulting representations improve performance over current approaches for hypernym prediction and image-caption retrieval.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Experiments show significant improvements of the method over strong recent baselines both in terms of representational capacity and generalization, including on hypernymy link prediction for word embeddings over Nickel & Kiela (2017) and Vendrov et al. (2015).
“Experiments show significant improvements of our method over strong recent baselines both in terms of representational capacity and generalization.”
From Hyperbolic Entailment Cones for Learning Hierarchical Embeddings (2018) · cites this paper as Vendrov et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334 · see citing works
Flagged first, then by citations
Research works printed but not linked: 26
- Baroni, Marco, Bernardi, Raffaella, Do, Ngoc-Quynh, and Shan, Chung-chieh. Entailment above the word level in distributional semantics. In EACL , 2012.
- Bordes, Antoine, Weston, Jason, Collobert, Ronan, and Bengio, Yoshua. Learning structured embeddings of knowledge bases. In AAAI , 2011.
- Bowman, Samuel R., Angeli, Gabor, Potts, Christopher, and Manning, Christopher D. A large annotated corpus for learning natural language inference. In EMNLP , 2015.
- Cho, Kyunghyun, Van Merri¨ enboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoderdecoder for statistical machine translation. In EMNLP , 2014.
- Chopra, Sumit, Hadsell, Raia, and LeCun, Yann. Learning a similarity metric discriminatively, with application to face verification. In CVPR , 2005.
- Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.
- Frome, Andrea, Corrado, Greg S, Shlens, Jon, Bengio, Samy, Dean, Jeff, Mikolov, Tomas, et al. Devise: A deep visual-semantic embedding model. In NIPS , 2013.
- Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. In ICLR , 2015.
- Kiros, Ryan, Salakhutdinov, Ruslan, and Zemel, Richard S. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 , 2014. · arXiv
- Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S, Torralba, Antonio, Urtasun, Raquel, and Fidler, Sanja. Skip-thought vectors. NIPS , 2015.
- Klein, Benjamin, Lev, Guy, Sadeh, Gil, and Wolf, Lior. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR , 2015.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.
- Lin, Dahua, Fidler, Sanja, Kong, Chen, and Urtasun, Raquel. Visual semantic search: Retrieving videos via complex textual queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2014a.
- Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Doll´ ar, Piotr, and Zitnick, C Lawrence. Microsoft coco: Common objects in context. In ECCV , 2014b.
- Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, and Yuille, Alan. Deep captioning with multimodal recurrent neural networks (m-rnn). In ICLR , 2015.
- Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey. Linguistic regularities in continuous space word representations. In HLT-NAACL , pp. 746-751, 2013.
- Miller, George A. Wordnet: a lexical database for english. Communications of the ACM , 1995.
- Norouzi, Mohammad, Mikolov, Tomas, Bengio, Samy, Singer, Yoram, Shlens, Jonathon, Frome, Andrea, Corrado, Greg S, and Dean, Jeffrey. Zero-shot learning by convex combination of semantic embeddings. In ICLR , 2014.
- Plummer, Bryan, Wang, Liwei, Cervantes, Chris, Caicedo, Juan, Hockenmaier, Julia, and Lazebnik, Svetlana. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-tosentence models. arXiv preprint arXiv:1505.04870 , 2015. · arXiv
- Rockt¨ aschel, Tim, Grefenstette, Edward, Hermann, Karl Moritz, Koˇ cisk` y, Tom´ aˇ s, and Blunsom, Phil. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664 , 2015. · arXiv
- Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.
- Socher, Richard, Chen, Danqi, Manning, Christopher D, and Ng, Andrew. Reasoning with neural tensor networks for knowledge base completion. In NIPS , 2013.
- Socher, Richard, Karpathy, Andrej, Le, Quoc V, Manning, Christopher D, and Ng, Andrew Y. Grounded compositional semantics for finding and describing images with sentences. TACL , 2014.
- Van der Maaten, Laurens and Hinton, Geoffrey. Visualizing data using t-sne. Journal of Machine Learning Research , 9(2579-2605):85, 2008.
- Vilnis, Luke and McCallum, Andrew. Word representations via gaussian embedding. In ICLR , 2015.
- Young, Peter, Lai, Alice, Hodosh, Micah, and Hockenmaier, Julia. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL , 2014.
Cited by
Sorted by citations
A Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743 · see citing works
“…encoders (Bowman et al., 2016) 83.2% 300D NTI-SLSTM-LSTM encoders (Munkhdalai & Yu, 2016a) 83.4% 1024D GRU encoders with SkipThoughts pre-training (Vendrov et al., 2015) 81.4% 300D NSE encoders (Munkhdalai & Yu, 2016b) 84.6% Our method 84.4% We find that compared to other published approaches, our…”
ResultInfluentialStacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works
DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., & Zhang, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11941 · Cited by 779 · see citing works
Poincaré Embeddings for Learning Hierarchical Representations
Nickel, M., & Kiela, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08039 · Cited by 667 · see citing works
Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings
Ren, H., Hu, W., & Leskovec, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05969 · Cited by 135 · see citing works
Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
Ganea, O.-E., Bécigneul, G., & Hofmann, T. (2018) · arXiv · doi:10.48550/arxiv.1804.01882 · Cited by 115 · see citing works
“We compare against the strong hierarchical embedding methods of Order embeddings (Vendrov et al., 2015) and Poincaré embeddings (Nickel & Kiela, 2017).”
BackgroundMethodInfluentialProbabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning
Chen, X., Boratko, M., Chen, M., Dasgupta, S. S., Li, X. L., & McCallum, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.68 · Cited by 47 · see citing works
“embeddings (Vendrov et al., 2016) take a regionbased approach, representing nodes of a graph using infinite cones, and using containment between cones to represent edges.”
BackgroundModeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
Bai, Y., Ying, R., Ren, H., & Leskovec, J. (2021) · arXiv · doi:10.48550/arxiv.2110.14923 · Cited by 31 · see citing works
“The most related line of work is learning structured embeddings to perform hierarchical reasoning on graphs and ontologies: order embedding, probabilistic order embedding, box embedding, Gumbel-box embedding and hyperbolic embedding [10, 11, 12, 19, 20, 21, 22].”
BackgroundMethodInfluentialBox-To-Box Transformations for Modeling Joint Hierarchies
Dasgupta, S. S., Li, X. L., Boratko, M., Zhang, D., & McCallum, A. (2021) · Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021) · doi:10.18653/v1/2021.repl4nlp-1.28 · Cited by 2 · see citing works
- Authors
- Vendrov, I., Kiros, R., Fidler, S., & Urtasun, R.
- Affiliations
- University of Toronto (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1511.06361
- OpenAlex
- W2172888184
- Cited by
- 296
- References
- 5
- Citations by year
- 2025: 5 · 2024: 15 · 2023: 15 · 2022: 22 · 2021: 28 · 2020: 52 · 2019: 63 · 2018: 51 · 2017: 28 · 2016: 15 · 2015: 1 · 2012: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- order embeddings, partially ordered sets, natural language inference, image captioning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
