From General to Specific: Informative Scene Graph Generation via Balance Adjustment
Open accessAbstract
The scene graph generation (SGG) task aims to detect visual relationship triplets, i.e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding. However, current models are stuck in common predicates, e.g., "on" and "at", rather than informative ones, e.g., "standing on" and "looking at", resulting in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "blocking" to describe an image, it is easy to misunderstand the scene. We argue that this phenomenon is caused by two key imbalances between informative predicates and common ones, i.e., semantic space level imbalance and training sample level imbalance. To tackle this problem, we propose BA-SGG, a simple yet effective SGG framework based on balance adjustment but not the conventional distribution fitting. It integrates two components: Semantic Adjustment (SA) and Balanced Predicate Learning (BPL), respectively for adjusting these imbalances. Benefited from the model-agnostic process, our method is easily applied to the state-of-the-art SGG models and significantly improves the SGG performance. Our method achieves 14.3%, 8.0%, and 6.1% higher Mean Recall (mR) than that of the Transformer model at three scene graph generation sub-tasks on Visual Genome, respectively. Codes are publicly available1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A large amount of recent attempts study how to use as unified as possible pipelines to handle various modalities and multimodal tasks.
“A large amount of recent attempts [117], [296], [297], [298] study how to use as unified as possible pipelines to handle various modalities and multimodal tasks.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [297]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Term-Weighting Approaches in Automatic Text Retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 11,421 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Region-based convolutional networks for accurate object detection and segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2437384 · Cited by 2,912 · see citing works
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271 · see citing works
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,118 · see citing works
Neural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00611 · Cited by 1,095 · see citing works
Unbiased Scene Graph Generation From Biased Training
Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00377 · Cited by 784 · see citing works
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 752 · see citing works
Visual Translation Embedding Network for Visual Relation Detection
Zhang, H., Kyaw, Z., Chang, S.-F., & Chua, T.-S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.331 · Cited by 586 · see citing works
Learning to Compose Dynamic Tree Structures for Visual Contexts
Tang, K., Zhang, H., Wu, B., Luo, W., & Liu, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00678 · Cited by 558 · see citing works
Destruction and Construction Learning for Fine-Grained Image Recognition
Chen, Y., Bai, Y., Zhang, W., & Mei, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00530 · Cited by 550 · see citing works
Learning a Discriminative Filter Bank Within a CNN for Fine-Grained Recognition
Wang, Y., Morariu, V. I., & Davis, L. S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00436 · Cited by 451 · see citing works
Knowledge-Embedded Routing Network for Scene Graph Generation
Chen, T., Yu, W., Chen, R., & Lin, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00632 · Cited by 444 · see citing works
Scene Graph Generation With External Knowledge and Image Reconstruction
Gu, J., Zhao, H., Lin, Z., Li, S., Cai, J., & Ling, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00207 · Cited by 315 · see citing works
GPS-Net: Graph Property Sensing Network for Scene Graph Generation
Lin, X., Ding, C., Zeng, J., & Tao, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00380 · Cited by 297 · see citing works
Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering
Li, X., Song, J., Gao, L., Liu, X., Huang, W., He, X., & Gan, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018658 · Cited by 265 · see citing works
Self-citationBridging Knowledge Graphs to Generate Scene Graphs
Zareian, A., Karaman, S., & Chang, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58592-1_36 · Cited by 212 · see citing works
Pixels to Graphs by Associative EmbeddingPreprint
Newell, A., & Deng, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.07365 · Cited by 204 · see citing works
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J., & Loy, C. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_20 · Cited by 155 · see citing works
Self-citationFine-Grained Recognition as HSnet Search for Informative Image Parts
Lam, M., Mahasseni, B., & Todorović, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.688 · Cited by 142 · see citing works
Measuring Prerequisite Relations Among Concepts
Liang, C., Wu, Z., Huang, W., & Giles, C. L. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1193 · Cited by 126 · see citing works
Mapping Images to Scene Graphs with Permutation-Invariant Structured PredictionPreprint
Herzig, R., Raboh, M., Chechik, G., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05451 · Cited by 101 · see citing works
Scene Graph Generation by Iterative Message PassingPreprint
Xu, D., Zhu, Y., Choy, C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.02426 · Cited by 100 · see citing works
Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation
Hung, Z.-S., Mallya, A., & Lazebnik, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992222 · Cited by 89 · see citing works
VrR-VG: Refocusing Visually-Relevant Relationships
Liang, Y., Bai, Y., Zhang, W., Qian, X., Zhu, L., & Mei, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01050 · Cited by 77 · see citing works
From Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Song, J., Zeng, P., Gao, L., & Shen, H. T. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/126 · Cited by 72 · see citing works
Self-citationHierarchical Graph Attention Network for Visual Relationship Detection
Li, M., & Chen, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01390 · Cited by 71 · see citing works
Pragmatically Informative Image Captioning with Character-Level Inference
Cohn-Gordon, R., Goodman, N. D., & Potts, C. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2070 · Cited by 62 · see citing works
Visual Translation Embedding Network for Visual Relation DetectionPreprint
Zhang, H., Kyaw, Z., Chang, S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08319 · Cited by 61 · see citing works
Knowledge-Embedded Routing Network for Scene Graph GenerationPreprint
Chen, T., Yu, W., Chen, R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03326 · Cited by 45 · see citing works
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph GenerationPreprint
Li, Y., Ouyang, W., Zhou, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.11538 · Cited by 44 · see citing works
Informative Image Captioning with External Sources of Information
Zhao, S., Sharma, P. K., Levinboim, T., & Soricut, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1650 · Cited by 39 · see citing works
One-shot Scene Graph Generation
Guo, Y., Song, J., Gao, L., & Shen, H. T. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414025 · Cited by 27 · see citing works
Self-citationZoom-Net: Mining Deep Feature Interactions for Visual Relationship RecognitionPreprint
Yin, G., Sheng, L., Liu, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.04979 · Cited by 16 · see citing works
One-shot Scene Graph GenerationPreprint
Guo, Y., Song, J., Gao, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.10824 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
- Authors
- Guo, Y., Gao, L., Wang, X., Hu, Y., Xu, X., Lu, X., Shen, H. T., & Song, J.
- Affiliations
- University of Electronic Science and Technology of China (CN)
- Southwest University (CN)
- Kuaishou,China
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 16363-16372
- OpenAlex
- W3196936439
- Cited by
- 97
- References
- 44
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 23 · 2023: 36 · 2022: 16 · 2021: 1
- FWCI
- 6.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- scene graph generation, visual relationship detection, scene understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
