The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Scene Graph Generation from Objects, Phrases and Region Captions
Open accessAbstract
Object detection, scene graph generation and region captioning, which are three scene understanding tasks at different semantic levels, are tied together: scene graphs are generated on top of objects detected in an image with their pairwise relationship predicted, while region captioning gives a language description of the objects, their attributes, relations and other context information. In this work, to leverage the mutual connections across semantic levels, we propose a novel neural network model, termed as Multi-level Scene Description Network (denoted as MSDN), to solve the three vision tasks jointly in an end-to-end manner. Object, phrase, and caption regions are first aligned with a dynamic graph based on their spatial and semantic connections. Then a feature refining structure is used to pass messages across the three levels of semantic tasks through the graph. We benchmark the learned model on three tasks, and show the joint learning across three tasks with our proposed method can bring mutual improvements over previous models. Particularly, on the scene graph generation task, our proposed method outperforms the stateof- art method with more than 3% margin. Code has been made publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
(2016) · Cited by 31,590
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
(2005) · Cited by 4,152
NLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10578-9_23 · Cited by 3,034
Facial Landmark Detection by Deep Multi-task Learning
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_7 · Cited by 1,365
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,204
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.494 · Cited by 1,137
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,069
NLTK: The Natural Language Toolkit
Bird, S. (2006) · Proceedings of the COLING/ACL on Interactive presentation sessions - · doi:10.3115/1225403.1225421 · Cited by 841
BabyTalk: Understanding and Generating Simple Image Descriptions
Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.162 · Cited by 839
Using Multiple Segmentations to Discover Objects and their Extent in Image Collections
Russell, B. C., Freeman, W. T., Efros, A. A., Sivic, J., & Zisserman, A. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.326 · Cited by 617
Visual Translation Embedding Network for Visual Relation Detection
Zhang, H., Kyaw, Z., Chang, S.-F., & Chua, T.-S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.331 · Cited by 586
Person Search with Natural Language Description
Li, S., Xiao, T., Li, H., Zhou, B., Yue, D., & Wang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.551 · Cited by 550
Self-citationDetecting Visual Relationships with Deep Relational Networks
Dai, B., Zhang, Y., & Lin, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.352 · Cited by 502
Multi-task learning for classification with dirichlet process priors
Xue, Y., Liao, X., Carin, L., Krishnapuram, B., & Bartlett, P. (2007) · Cited by 485
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Dai, B., Fidler, S., Urtasun, R., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.323 · Cited by 477
Barnard, K., Duygulu, P., Forsyth, D. A., de Freitas, N., Blei, D. M., Jordan, M. I., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2003) · Cited by 459
Harvey, D., & Moran, S. (2009) · Cited by 439
Recognition using visual phrases
Sadeghi, M. A., & Farhadi, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995711 · Cited by 417
A convex formulation for learning task relationships in multi-task learning
Zhang, Y., & Yeung, D. (2010) · Cited by 338
Learning with Whom to Share in Multi-task Feature Learning
Kang, Z., Grauman, K. L., & Sha, F. (2011) · Cited by 320
Simple Baseline for Visual Question Answering
Zhou, B., Tian, Y., Sukhbaatar, S., Szlam, A. D., & Fergus, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02167 · Cited by 254
Self-citationDeep Variation-Structured Reinforcement Learning for Visual Relationship and Attribute Detection
Liang, X., Lee, L., & Xing, E. P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.469 · Cited by 243
Multi-level Attention Networks for Visual Question Answering
Yu, D., Fu, J., Mei, T., & Rui, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.446 · Cited by 230
ViP-CNN: Visual Phrase Guided Convolutional Neural Network
Li, Y., Ouyang, W., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.766 · Cited by 229
Self-citationSocher, R., & Fei-Fei, L. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540112 · Cited by 217
Learning cross-modality similarity for multinomial data
Jia, Y., Salzmann, M., & Darrell, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126524 · Cited by 181
Understanding Indoor Scenes Using 3D Geometric Phrases
Choi, W., Chao, Y.-W., Pantofaru, C., & Savarese, S. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.12 · Cited by 176
Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers
Gupta, A., & Davis, L. S. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_3 · Cited by 167
Detecting Actions, Poses, and Objects with Relational Phraselets
Desai, C., & Ramanan, D. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33765-9_12 · Cited by 165
Learning a Recurrent Visual Representation for Image Caption Generation
Chen, X., & Zitnick, C. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5654 · Cited by 160
PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
Zhang, H., Kyaw, Z., Yu, J., & Chang, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.454 · Cited by 136
A Convex Formulation for Learning Task Relationships in Multi-Task LearningPreprint
Zhang, Y., & Yeung, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.3536 · Cited by 130
Efficiently selecting regions for scene understanding
Kumar, M., & Koller, D. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540072 · Cited by 128
Probabilistic Multi-Task Feature Selection
Zhang, Y., Yeung, D., & Xu, Q. (2010) · Cited by 83
Generalizing Image Captions for Image-Text Parallel Corpus
Кузнецова, П., Ordóñez, V., Berg, A., Berg, T. L., & Choi, Y. (2013) · Cited by 73
Care about you: towards large-scale human-centric visual relationship detectionPreprint
Zhuang, B., Wu, Q., Shen, C., Reid, I. D., & van den Hengel, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09892 · Cited by 27
Towards Context-aware Interaction RecognitionPreprint
Zhuang, B., Liu, L., Shen, C., & Reid, I. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06246 · Cited by 9
Cited by
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2018) · arXiv · doi:10.48550/arxiv.1809.02165 · Cited by 2,965
Self-citationDeep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01247-4 · Cited by 2,965
Self-citationNeural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00611 · Cited by 1,052
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_42 · Cited by 870
Self-citationGraph R-CNN for Scene Graph Generation
Yang, J., Lu, J., Lee, S., Batra, D., & Parikh, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_41 · Cited by 851
Metadata
- Authors
- Li, Y., Ouyang, W., Zhou, B., Wang, K., & Wang, X.
- Venue
- 2017 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2017-10
- Pages
- 1270-1279
- OpenAlex
- W2963649796
- Cited by
- 500
- References
- 53
- FWCI
- 15.29
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.