Learning to Generate Scene Graph from Natural Language Supervision
Open accessAbstract
Learning from image-text data has demonstrated recent success for many recognition tasks, yet is currently limited to visual features or individual visual concepts such as objects. In this paper, we propose one of the first methods that learn from image-sentence pairs to extract a graphical representation of localized objects and their relationships within an image, known as scene graph. To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph. Further, we design a Transformer-based model to predict these "pseudo" labels via a masked token prediction task. Learning from only image-sentence pairs, our model achieves 30% relative gain over a latest method trained with human-annotated unlocalized scene graphs. Our model also shows strong results for weakly and fully supervised scene graph generation. In addition, we explore an open-vocabulary setting for detecting scene graphs, and present the first result for open-set scene graph generation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multi-task training (by adding auxiliary loss) and adversarial training improve multimodal pretraining Transformers to further boost performance.
“Some practices demonstrate that multi-task training (by adding auxiliary loss) [111], [137] and adversarial training [210] improve multimodal pretraining Transformers to further boost the performance.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [210]
VILLA is a two-stage strategy (task-agnostic adversarial pretraining followed by task-specific adversarial finetuning) that improves VLP Transformers.
“VILLA [210] is a two-stage strategy (taskagnostic adversarial pretraining, followed by task-specific adversarial finetuning) that improves VLP Transformers.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [210]
VILLA is a generic adversarial training framework that can be applied to various multimodal Transformers.
“VILLA [210] is a generic adversarial training framework that can be applied to various multimodal Transformers.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [210]
The citation lists haven’t been fetched yet
References 75 · Cited by 73
- Authors
- Zhong, Y., Shi, J., Yang, J., Xu, C., & Li, Y.
- Affiliations
- University of Wisconsin–Madison (US)
- University of Rochester (US)
- Microsoft (United States) (US)
- University of Wisconsin-Madison
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 1803-1814
- OpenAlex
- W3198299822
- Cited by
- 73
- References
- 75
- Citations by year
- 2026: 11 · 2025: 7 · 2024: 15 · 2023: 26 · 2022: 13 · 2021: 1
- FWCI
- 4.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- scene graph generation, image-text pairs, pseudo-labeling, object detection, masked token prediction, transformer models, weakly supervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
