The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval
Open accessAbstract
Semantically complex queries which include attributes of objects and relations between objects still pose a major challenge to image retrieval systems.Recent work in computer vision has shown that a graph-based semantic representation called a scene graph is an effective representation for very detailed image descriptions and for complex queries for retrieval.In this paper, we show that scene graphs can be effectively created automatically from a natural language scene description.We present a rule-based and a classifierbased scene graph parser whose output can be used for image retrieval.We show that including relations and attributes in the query graph outperforms a model that only considers objects and that using the output of our parsers is almost as effective as using human-constructed scene graphs (Recall@10 of 27.1% vs. 33.4%).Additionally, we demonstrate the general usefulness of parsing to scene graphs by showing that the output can also be used to generate 3D scenes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
Self-citationA survey of content-based image retrieval with high-level semantics
Liu, Y., Zhang, D., Lu, G., & Ma, W.-Y. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.04.045 · Cited by 1,696 · cited 2×
“Rui et al. (1999), Liu et al. (2007), and Siddiquie et al. (2011) provide overviews of the developments of this field over the last twenty years.”
BackgroundImage Retrieval: Current Techniques, Promising Directions, and Open Issues
Rui, Y., Huang, T. S., & Chang, S.-F. (1999) · Journal of Visual Communication and Image Representation · doi:10.1006/jvci.1999.0413 · Cited by 1,601
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
(2013) · Cited by 1,117
Image retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101
Self-citationGrounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 777
Self-citationDeep Fragment Embeddings for Bidirectional Image Sentence Mapping
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 739
Self-citationDeep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 620
Hobbs, J. R. (1978) · Lingua · doi:10.1016/0024-3841(78)90006-2 · Cited by 612
Universal Stanford dependencies: A cross-linguistic typology
de Marneffe, M.-C., Dozat, T., Silveira, N., Haverinen, K., Ginter, F., Nivre, J., & Manning, C. D. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4ok3zdjx8t96 · Cited by 456
Self-citationDeterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules
Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., & Jurafsky, D. (2013) · Computational Linguistics · doi:10.1162/coli_a_00152 · Cited by 384
Self-citationImage ranking and retrieval based on multi-attribute queries
Siddiquie, B., Feris, R. S., & Davis, L. S. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995329 · Cited by 313
A Discriminative Graph-Based Parser for the Abstract Meaning Representation
Flanigan, J., Thomson, S., Carbonell, J., Dyer, C., & Smith, N. A. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-1134 · Cited by 311
Smatch: an Evaluation Metric for Semantic Feature Structures
Cai, S., & Knight, K. K. (2013) · Cited by 281
The New Data and New Challenges in Multimedia Research.Preprint
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv (Cornell University) · Cited by 247
Learning Spatial Knowledge for Text to 3D Scene Generation
Chang, A., Savva, M., & Manning, C. D. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1217 · Cited by 210
Self-citationA Transition-based Algorithm for AMR Parsing
Wang, C., Xue, N., & Pradhan, S. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1040 · Cited by 159
Visual Semantic Search: Retrieving Videos via Complex Textual Queries
Lin, D., Fidler, S., Kong, C., & Urtasun, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.340 · Cited by 158
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.211 · Cited by 152
YFCC100M: The New Data in Multimedia Research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv · doi:10.48550/arxiv.1503.01817 · Cited by 69
Learning language through pictures
Chrupała, G., Kádár, Á., & Alishahi, A. (2015) · doi:10.3115/v1/p15-2019 · Cited by 53
Robust Subgraph Generation Improves Abstract Meaning Representation Parsing
Werling, K., Angeli, G., & Manning, C. D. (2015) · doi:10.3115/v1/p15-1095 · Cited by 50
Self-citation
Cited by
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Self-citationSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,069
Self-citationA Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879
“PICE. SPICE (Semantic Propositional Image Caption Evaluation) [3] is a new caption evaluation metric based on semantic concept. It is based on a graph-based semantic representation called scene-graph [63, 123]. This graph can extract the information of different objects, attributes and their relationships from the ima…”
MethodImage Generation from Scene Graphs
Johnson, J., Gupta, A., & Fei-Fei, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00133 · Cited by 836
Self-citation
Metadata
- Authors
- Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., & Manning, C. D.
- Venue
- Proceedings of the Fourth Workshop on Vision and Language
- Published
- 2015
- Pages
- 70-80
- OpenAlex
- W2250378130
- Cited by
- 332
- References
- 28
- FWCI
- 6.37
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.