Scene Graph Generation by Iterative Message Passing
Open accessAbstract
Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. Our key insight is that the graph generation problem can be formulated as message passing between the primal node graph and its dual edge graph. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods on the Visual Genome dataset as well as support relation inference in NYU Depth V2 dataset.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Self-citationRegion-based convolutional networks for accurate object detection and segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2437384 · Cited by 2,912 · see citing works
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908 · see citing works
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 4× · see citing works
“As shown in previous work [19] and [43], dense graph inference can be approximated by mean field in Conditional Random Fields (CRF).”
BackgroundMethodInfluentialSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · see citing works
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
Bell, S., Zitnick, C. L., Bala, K., & Girshick, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.314 · Cited by 1,344 · see citing works
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133 · see citing works
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,118 · see citing works
Self-citationImage retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101 · see citing works
Self-citationThe role of context in object recognition
Oliva, A., & Torralba, A. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.009 · Cited by 1,063 · see citing works
Contextual Priming for Object Detection
Torralba, A. B. (2003) · International Journal of Computer Vision · doi:10.1023/a:1023052124951 · Cited by 777 · see citing works
Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., & Belongie, S. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408986 · Cited by 608 · see citing works
Modeling mutual context of object and human pose in human-object interaction activities
Yao, B., & Fei-Fei, L. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540235 · Cited by 574 · see citing works
Self-citationObject categorization using co-occurrence, location and appearance
Galleguillos, C., Rabinovich, A., & Belongie, S. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587799 · Cited by 467 · see citing works
Graph-Structured Representations for Visual Question Answering
Teney, D., Liu, L., & van den Hengel, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.344 · Cited by 464 · see citing works
Recognition using visual phrases
Sadeghi, M. A., & Farhadi, A. (2011) · doi:10.1109/cvpr.2011.5995711 · Cited by 421 · see citing works
Semantic Object Parsing with Graph LSTM
Liang, X., Shen, X., Feng, J., Lin, L., & Yan, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_8 · Cited by 368 · see citing works
Learning to share visual appearance for multiclass object detection
Salakhutdinov, R., Torralba, A. B., & Tenenbaum, J. B. (2011) · doi:10.1109/cvpr.2011.5995720 · Cited by 326 · see citing works
Graph Cut Based Inference with Co-occurrence Statistics
Ladický, Ľ., Russell, C., Kohli, P., & Torr, P. H. S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15555-0_18 · Cited by 307 · see citing works
HICO: A Benchmark for Recognizing Human-Object Interactions in Images
Chao, Y.-W., Wang, Z., He, Y., Wang, J., & Deng, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.122 · Cited by 300 · see citing works
Holistic Scene Understanding for 3D Object Detection with RGBD Cameras
Lin, D., Fidler, S., & Urtasun, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.179 · Cited by 253 · see citing works
Situation Recognition: Visual Semantic Role Labeling for Image Understanding
Yatskar, M., Zettlemoyer, L., & Farhadi, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.597 · Cited by 253 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsPreprint
Krishna, R., Zhu, Y., Groth, O., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07332 · Cited by 248 · see citing works
Self-citationDiscriminative Models for Multi-Class Object Layout
Desai, C., Ramanan, D., & Fowlkes, C. C. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0439-x · Cited by 208 · see citing works
Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers
Gupta, A. K., & Davis, L. S. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88682-2_3 · Cited by 180 · see citing works
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.211 · Cited by 156 · see citing works
Characterizing structural relationships in scenes using graph kernels
Fisher, M., Savva, M., & Hanrahan, P. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964929 · Cited by 154 · see citing works
Learning Spatial Knowledge for Text to 3D Scene Generation
Chang, A., Savva, M., & Manning, C. D. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1217 · Cited by 154 · see citing works
Scene Parsing by Integrating Function, Geometry and Appearance Models
Zhao, Y., & Zhu, S.-C. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.401 · Cited by 135 · see citing works
Discriminative models for static human-object interactions
Desai, C., Ramanan, D., & Fowlkes, C. C. (2010) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2010.5543176 · Cited by 112 · see citing works
Learning semantic relationships for better action retrieval in images
Ramanathan, V., Li, C., Deng, J., Han, W., Li, Z., Gu, K., Song, Y., Samy Bengio, A., Rosenberg, C., & Fei-Fei, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298713 · Cited by 109 · see citing works
Self-citation3D-Based Reasoning with Blocks, Support, and Stability
Jia, Z., Gallagher, A., Saxena, A., & Chen, T. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.8 · Cited by 94 · see citing works
Scene Understanding by Reasoning Stability and Safety
Zheng, B., Zhao, Y., Yu, J. C., Ikeuchi, K., & Zhu, S. C. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0795-4 · Cited by 59 · see citing works
On support relations and semantic scene graphs
Yang, M. Y., Liao, W., Ackermann, H., & Rosenhahn, B. (2017) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.07.010 · Cited by 48 · see citing works
Describing Common Human Visual Actions in Images
Ronchi, M. R., & Perona, P. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.52 · Cited by 32 · see citing works
Statistics of 3D Object Locations in Images
Baur, R., Efros, A. A., & Hebert, M. (2018) · doi:10.1184/r1/6560990 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
Graph convolutional networks: a comprehensive review
Zhang, S., Tong, H., Xu, J., & Maciejewski, R. (2019) · Computational Social Networks · doi:10.1186/s40649-019-0069-y · Cited by 1,888 · see citing works
Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs
Landrieu, L., & Simonovsky, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00479 · Cited by 1,543 · see citing works
Image Generation from Scene Graphs
Johnson, J., Gupta, A., & Fei-Fei, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00133 · Cited by 836 · see citing works
Self-citationIterative Visual Reasoning Beyond Convolutions
Chen, X., Li, L.-J., Fei-Fei, L., & Gupta, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00756 · Cited by 236 · see citing works
Self-citationFrom General to Specific: Informative Scene Graph Generation via Balance Adjustment
Guo, Y., Gao, L., Wang, X., Hu, Y., Xu, X., Lu, X., Shen, H. T., & Song, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01607 · Cited by 97 · see citing works
Explainable goal-driven agents and robots-a comprehensive review
Sado, F., Loo, C. K., Liew, W. S., Kerzel, M., & Wermter, S. (2023) · ACM Computing Surveys · doi:10.1145/3564240 · Cited by 79 · see citing works
“[224] to generate structured scene representations from input images.”
BackgroundLearning to Generate Scene Graph from Natural Language Supervision
Zhong, Y., Shi, J., Yang, J., Xu, C., & Li, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00184 · Cited by 73 · see citing works
Target Adaptive Context Aggregation for Video Scene Graph Generation
Teng, Y., Wang, L., Li, Z., & Wu, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01343 · Cited by 74 · see citing works
Visual Semantic Information Pursuit: A Survey
Liu, D., Bober, M., & Kittler, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2950025 · Cited by 39 · see citing works
Visual Question Generation: The State of the Art
Patil, C., & Patwardhan, M. (2021) · ACM Computing Surveys · doi:10.1145/3383465 · Cited by 25 · see citing works
Multimodal Generative Models for Compositional Representation Learning
Wu, M., & Goodman, N. (2019) · arXiv · doi:10.48550/arxiv.1912.05075 · Cited by 8 · see citing works
- Authors
- Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 3097-3106
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2579549467
- Cited by
- 1,271
- References
- 45
- Citations by year
- 2026: 74 · 2025: 126 · 2024: 164 · 2023: 183 · 2022: 171 · 2021: 206 · 2020: 158 · 2019: 131 · 2018: 52 · 2017: 6
- FWCI
- 27.74
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Multidisciplinary University Research Initiative
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- scene graph generation, message passing, object relations, visual grounding, graph generation, dual graph, end-to-end learning, Visual Genome, NYU Depth V2
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
