Image retrieval using scene graphs
The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
This paper develops a novel framework for semantic image retrieval based on the notion of a scene graph. Our scene graphs represent objects (“man”, “boat”), attributes of objects (“boat is white”) and relationships between objects (“man standing on boat”). We use these scene graphs as queries to retrieve semantically related images. To this end, we design a conditional random field model that reasons about possible groundings of scene graphs to test images. The likelihoods of these groundings are used as ranking scores for retrieval. We introduce a novel dataset of 5,000 human-generated scene graphs grounded to images and use this dataset to evaluate our method for image retrieval. In particular, we evaluate retrieval using full scene graphs and small scene subgraphs, and show that our method outperforms retrieval methods that use only objects or low-level image features. In addition, we show that our full model can be used to improve object localization compared to baseline methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 1×
“Datasets have been a driving force of computer vision algorithms, ranging from classi-fication [64] to object recognition and segmentation [18, 12, 55, 15].”
MethodAre we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,384
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,319
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
LabelMe: A Database and Web-Based Tool for Image Annotation
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0090-8 · Cited by 4,210
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283
Product Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · cited 2×
“We adapt the metrics from [27]; specifically, for each method we report the median rank of the highest ranked true positive image I 2 , . . . Fig.”
MethodFei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Self-citationYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206772 · Cited by 1,917
Attribute-Based Classification for Zero-Shot Visual Object Categorization
Lampert, C. H., Nickisch, H., & Harmeling, S. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.140 · Cited by 1,627
Aggregating Local Image Descriptors into Compact Codes
Jegou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., & Schmid, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.235 · Cited by 1,503
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · cited 1×
“More recently, there has been a shift away from iconic images toward datasets depicting objects in the context of entire scenes, e.g., SUN09 [9], PASCAL-Context [46], and the COCO [42] datasets.”
BackgroundFraming Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,314
Measuring the Objectness of Image Windows
Alexe, B., Deselaers, T., & Ferrari, V. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.28 · Cited by 1,186
Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval
Chum, O., Philbin, J., Sivic, J., Isard, M., & Zisserman, A. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408891 · Cited by 859
Parikh, D., & Grauman, K. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126281 · Cited by 851
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 777
Large-scale image retrieval with compressed Fisher vectors
Perronnin, F., Liu, Y., Sanchez, J., & Poirier, H. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540009 · Cited by 761
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 739
Self-citationMidge: Generating Image Descriptions From Computer Vision Detections
Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T. L., & Daumé, H. (2012) · Conference of the European Chapter of the Association for Computational Linguistics · Cited by 546
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995466 · Cited by 521
Stanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task
Lee, H., Peirsman, Y., Chang, A. L. S., Chambers, N., Surdeanu, M., & Jurafsky, D. (2011) · Lirias · Cited by 504
Li, L.-J., Socher, R., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206718 · Cited by 484
Self-citationFerrari, V., & Zisserman, A. (2007) · neural information processing systems · Cited by 431
Recognition using visual phrases
Sadeghi, M. A., & Farhadi, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995711 · Cited by 417
Automatic Attribute Discovery and Characterization from Noisy Web Data
Berg, T. L., Berg, A. C., & Shih, J. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_48 · Cited by 411
Efficient Object Category Recognition Using Classemes
Torresani, L., Szummer, M., & Fitzgibbon, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_56 · Cited by 409
NEIL: Extracting Visual Knowledge from Web Data
Chen, X., Shrivastava, A., & Gupta, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.178 · Cited by 408
Krähenbühl, P., & Koltun, V. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_47 · Cited by 336
A Multi-Pass Sieve for Coreference Resolution
Raghunathan, K., Lee, H., Rangarajan, S., Chambers, N., Surdeanu, M., Jurafsky, D., & Manning, C. D. (2010) · Cited by 335
Crowdsourcing annotations for visual object detection
Su, H., Deng, J., & Fei-Fei, L. (2012) · National Conference on Artificial Intelligence · Cited by 328
Self-citationExploiting hierarchical context on a large database of object categories
Choi, M. J., Lim, J. J., Torralba, A., & Willsky, A. S. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540221 · Cited by 304
Learning Everything about Anything: Webly-Supervised Visual Concept Learning
Divvala, S. K., Farhadi, A., & Guestrin, C. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.412 · Cited by 267
The New Data and New Challenges in Multimedia Research.Preprint
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv (Cornell University) · Cited by 247
Self-citationSocher, R., & Fei-Fei, L. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540112 · Cited by 217
Self-citationCao, Y., Wang, C., Li, Z., Zhang, L., & Zhang, L. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540021 · Cited by 208
Discriminative Models for Multi-Class Object Layout
Desai, C., Ramanan, D., & Fowlkes, C. C. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0439-x · Cited by 208
Bringing Semantics into Focus Using Visual Abstraction
Zitnick, C. L., & Parikh, D. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.387 · Cited by 193
COSTA: Co-Occurrence Statistics for Zero-Shot Classification
Mensink, T., Gavves, E., & Snoek, C. G. M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.313 · Cited by 193
Yao, B., Yang, X., & Zhu, S.-C. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74198-5_14 · Cited by 188
Attribute-centric recognition for cross-category generalization
Farhadi, A., Endres, I., & Hoiem, D. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539924 · Cited by 188
Characterizing structural relationships in scenes using graph kernels
Fisher, M., Savva, M., & Hanrahan, P. (2011) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1964921.1964929 · Cited by 184
Understanding Indoor Scenes Using 3D Geometric Phrases
Choi, W., Chao, Y.-W., Pantofaru, C., & Savarese, S. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.12 · Cited by 176
Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers
Gupta, A., & Davis, L. S. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_3 · Cited by 167
DeepDive: Web-scale Knowledge-base Construction using Statistical Learning and Inference
Niu, F., Zhang, C., Ré, C., & SHAVLIK, J. W. (2012) · Cited by 164
Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World
Krishnamurthy, J., & Kollar, T. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00220 · Cited by 156
Context Driven Scene Parsing with Attention to Rare Classes
Yang, J., Price, B., Cohen, S., & Yang, M.-H. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.415 · Cited by 154
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.211 · Cited by 153
Learning Spatial Knowledge for Text to 3D Scene Generation
Chang, A., Savva, M., & Manning, C. D. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1217 · Cited by 153
Visual Semantic Search: Retrieving Videos via Complex Textual Queries
Lin, D., Fidler, S., Kong, C., & Urtasun, R. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.340 · Cited by 140
Scene Parsing by Integrating Function, Geometry and Appearance Models
Zhao, Y., & Zhu, S.-C. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.401 · Cited by 135
PiCoDes: Learning a Compact Code for Novel-Category Recognition
Bergamo, A., Torresani, L., & Fitzgibbon, A. W. (2011) · neural information processing systems · Cited by 114
Query Adaptive Similarity for Large Scale Object Retrieval
Qin, D., Wengert, C., & Gool, L. V. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.211 · Cited by 83
OpenGM: A C++ Library for Discrete Graphical ModelsPreprint
Andres, B., Beier, T., & Kappes, J. H. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.0111 · Cited by 79
Predicting Object Dynamics in Scenes
Fouhey, D. F., & Zitnick, C. L. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.260 · Cited by 73
Detecting objects in large image collections and videos by efficient subimage retrieval
Lampert, C. H. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459359 · Cited by 68
Efficient indexing for large scale visual search
Zhang, X., Li, Z., Zhang, L., Ma, W., & Shum, H. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459354 · Cited by 60
Size Matters: Exhaustive Geometric Verification for Image Retrieval
Stewénius, H., Gunderson, S. H., & Pilet, J. (2012) · Cited by 24
Research works printed but not linked: 5
- jurafsky, 2000, Speech & language processing
- lampert, 2013, Attribute-based classification for zero-shot learning of object categories, PAMI
- yao, 2011, Classifying actions and measuring action similarity by modeling the mutual context of objects and human poses, a) A 1(D2) D3
- 0, OpenSceneGraph
- 10.1145/1964921.1964929 · DOI
Cited by
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Self-citationPast, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I. D., & Leonard, J. J. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-138892 · Cited by 3,048
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Self-citation
Metadata
- Authors
- Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- Max Planck Institute for Informatics (DE)
- Yahoo (United States) (US)
- ORCID
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3668-3678
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2077069816
- Cited by
- 1,101
- References
- 68
- Citations by year
- 2026: 59 · 2025: 84 · 2024: 133 · 2023: 134 · 2022: 130 · 2021: 139 · 2020: 145 · 2019: 117 · 2018: 66 · 2017: 47 · 2016: 33 · 2015: 13
- FWCI
- 21.74
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- semantic image retrieval, scene graph, conditional random fields, object relations, image ranking, object localization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.