CoChat
Sign Up
Cited by 586Multimodal Machine Learning ApplicationsAdvanced Image and Video Retrieval TechniquesAdvanced Neural Network Applications

Visual Translation Embedding Network for Visual Relation Detection

Open access
Zhang, Hanwang, Kyaw, Zawlin, Chang, Shih-Fu, Chua, Tat-Seng·2017·2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2017.331
Open this paper in CoChat — free accountOpen PDF

Abstract

Visual relations, such as person ride bike and bike next to car, offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate ≈ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-toend relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu's multi-modal model with language priors [27].

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2017.331

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Viewing SimplE as a single-vector-per-entity model makes it easily integrable (or compatible) with other embedding models in knowledge graph completion, computer vision and natural language processing.

    “Viewing SimplE as a single-vector-per-entity model makes it easily integrable (or compatible) with other embedding models (in knowledge graph completion, computer vision and natural language processing) such as [35, 47, 36].”

    From SimplE Embedding for Link Prediction in Knowledge Graphs (2018) · cites this paper as [47]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label- embedding for image classification. TPAMI, 2015. 3
  2. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Deep composi- tional question answering with neural module networks. In CVPR, 2016. 2
  3. Y. Atzmon, J. Berant, V. Kezami, A. Globerson, and G. Chechik. Learning to generalize to new compositions in image understanding. In EMNLP, 2016. 2, 3
  4. A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi- relational data. In NIPS, 2013. 2, 3
  5. C. Desai, D. Ramanan, and C. C. Fowlkes. Discriminative models for multi-class object layout. IJCV, 2011. 2
  6. L. Dong, F. Wei, M. Zhou, and K. Xu. Question answering over freebase with multi-column convolutional neural networks. In ACL, 2015. 2
  7. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Gen- erating sentences from images. In ECCV, 2010. 2, 6
  8. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov, et al. Devise: A deep visual-semantic embedding model. In NIPS, 2013. 3
  9. R. Girshick. Fast r-cnn. In ICCV, 2015. 2, 3
  10. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hier- archies for accurate object detection and semantic segmentation. In CVPR, 2014. 3, 4, 7
  11. K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wier- stra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015. 2, 3 · arXiv
  12. A. Gupta and L. S. Davis. Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In ECCV, 2008. 2
  13. A. Gupta, A. Kembhavi, and L. S. Davis. Observing human-object interactions: Using spatial and functional compatibility for recogni- tion. TPAMI, 2009. 2
  14. A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. 1
  15. M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In NIPS, 2015. 2, 3, 4
  16. J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully convolu- tional localization networks for dense captioning. In CVPR, 2016. 3, 7
  17. J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. 2, 5, 7
  18. Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu. Learning entity and re- lation embeddings for knowledge graph completion. In AAAI, 2015. 2
  19. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed. Ssd: Single shot multibox detector. In ECCV, 2016. 3, 5
  20. C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei. Visual relationship detection with language priors. In ECCV, 2016. 1, 2, 3, 5, 6, 7, 8
  21. L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. JMLR, 2008. 5
  22. A. Mallya and S. Lazebnik. Learning models for actions and person- object interactions with transfer to question answering. In ECCV, 2016. 8
  23. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Dis- tributed representations of words and phrases and their composition- ality. In NIPS, 2013. 3, 7
  24. M. Nickel, K. Murphy, V. Tresp, and E. Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 2016. 3
  25. B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hock- enmaier, and S. Lazebnik. Flickr30k entities: Collecting region- to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015. 2
  26. V. Ramanathan, C. Li, J. Deng, W. Han, Z. Li, K. Gu, Y. Song, S. Bengio, C. Rossenberg, and L. Fei-Fei. Learning semantic re- lationships for better action retrieval in images. In CVPR, 2015. 2, 3, 6
  27. F. Sadeghi, S. K. Divvala, and A. Farhadi. Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. In CVPR, 2015. 2
  28. M. A. Sadeghi and A. Farhadi. Recognition using visual phrases. In CVPR, 2011. 2, 6, 7
  29. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 4 · arXiv
  30. L. Torresani, M. Szummer, and A. Fitzgibbon. Efficient object cate- gory recognition using classemes. In ECCV, 2010. 4
  31. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. TPAMI, 2016. 1
  32. P. Wang, Q. Wu, C. Shen, A. v. d. Hengel, and A. Dick. Fvqa: Fact- based visual question answering. arXiv preprint arXiv:1606.05433, 2016. 2 · arXiv
  33. B. Yao and L. Fei-Fei. Modeling mutual context of object and human pose in human-object interaction activities. In CVPR, 2010. 2

Cited by

Sorted by citations

Authors
Zhang, H., Kyaw, Z., Chang, S.-F., & Chua, T.-S.
Affiliations
  • Columbia University (US)
  • National University of Singapore (SG)
ORCID
Venue
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2017-07
Pages
3107-3115
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2017.331
OpenAlex
W2591644541
Cited by
586
References
9
Citations by year
2026: 25 · 2025: 32 · 2024: 35 · 2023: 65 · 2022: 72 · 2021: 89 · 2020: 87 · 2019: 97 · 2018: 58 · 2017: 26
FWCI
21.42
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Research Foundation Singapore
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
knowledge base embedding, scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review