The DOI matches its Crossref record; 118 of the 123 references it cites were found in a registry or an index.
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Open accessAbstract
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked “What vehicle is the person riding?”, computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) to answer correctly that “the person is riding a horse-drawn carriage.” In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 108K images where each image has an average of $$35$$ objects, $$26$$ attributes, and $$21$$ pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answer pairs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
Experimental — accuracy not yet measuredIn the citing paper
“For a few of these applications, some datasets were used like SQuAD, MS-MARCO and the like. Some works curated their own dataset owing to the distinctive application involved [122, 124].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
In the citing paper
“Also, most works used factoid-based questions but Yes/No [133], MCQ type [124] and reasoning type questions [122] were also generated for some applications.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in Ranjay Krishna Stanford University, Stanford, CA, USA E-mail: ranjaykrishna@cs.stanford.edu Yuke Zhu Stanford University, Stanford, CA, USA Oliver Groth Dresden University of…”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in Ranjay Krishna Stanford University, Stanford, CA, USA E-mail: ranjaykrishna@cs.stanford.edu Yuke Zhu Stanford University, Stanford, CA, USA Oliver Groth Dresden University of…”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“We then use the NeuralTalk (Karpathy and Fei-Fei, 2014) model to train a long short-term memory (LSTM) network (Hochreiter and Schmidhuber, 1997) to generate descriptions of regions.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 1
- Hayes, P. J. (1985). The second naive physics manifesto. Theories of the commonsense world (pp. 1–36).
Cited by
Sorted by citations
Items 501–600 of 1,032
Krishna, R., Chami, I., Bernstein, M. S., & Fei-Fei, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00718 · Cited by 98
Self-citationAmirian, S., Rasheed, K., Taha, T. R., & Arabnia, H. R. (2020) · IEEE Access · doi:10.1109/access.2020.3042484 · Cited by 103
Human-Centric Spatio-Temporal Video Grounding With Visual Transformers
Tang, Z., Liao, Y., Liu, S., Li, G., Jin, X., Jiang, H., Yu, Q., & Xu, D. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3085907 · Cited by 116
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering
Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y. N., & Natarajan, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00501 · Cited by 114
Trustworthy Long-Tailed Classification
Li, B., Han, Z., Li, H., Fu, H., & Zhang, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00684 · Cited by 101
Progressive Instance-Aware Feature Learning for Compositional Action Recognition
Yan, R., Xie, L., Shu, X., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261659 · Cited by 117
“However, it is difficult for these approaches to mine rich relationships [6], [12], [32], [33] between different instances (objects or hands or persons, etc.) in the spatial and temporal domains.”
BackgroundLogic Tensor Networks for Semantic Image Interpretation
Donadello, I., Serafini, L., & d'Avila Garcez, A. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/221 · Cited by 179
COCO Attributes: Attributes for People, Animals, and Objects
Patterson, G., & Hays, J. H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_6 · Cited by 97
Visual Relationship Detection With Deep Structural Ranking
Liang, K., Guo, Y., Chang, H., & Chen, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12274 · Cited by 97
Context Reasoning Attention Network for Image Super-Resolution
Zhang, Y., Wei, D., Qin, C., Wang, H., Pfister, H., & Fu, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00424 · Cited by 98
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 97
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 107
“…complex questions that requires deep reasoning R-VQA [116] 335k Instances of natural images, QA pairs, and relation facts from Visual Genome [85] Facts are filtered using a ranking algorithm and evaluated by humans To grasp the concept of counterexamples, let’s consider a dataset where…”
BackgroundMethodInfluentialMultiple Feature Integration for Classification of Thoracic Disease in Chest Radiography
Ho, T. K., & Gwak, J. (2019) · Applied Sciences · doi:10.3390/app9194130 · Cited by 96
Deep Metric Learning Beyond Binary Supervision
Kim, S., Seo, M., Laptev, I., Cho, M., & Kwak, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00239 · Cited by 102
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
Zhu, Y., Tremblay, J., Birchfield, S. T., & Zhu, Y. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561548 · Cited by 96
Self-citationSimple Multi-dataset Detection
Zhou, X., Koltun, V., & Krähenbühl, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00742 · Cited by 98
MIST : Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
Gao, D., Zhou, L., Ji, L., Zhu, L., Yang, Y., & Shou, M. Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01419 · Cited by 108
Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 130
Spatially Aware Multimodal Transformers for TextVQA
Kant, Y., Batra, D., Anderson, P., Schwing, A. G., Parikh, D., Lu, J., & Agrawal, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_41 · Cited by 95
High-Order Interaction Learning for Image Captioning
Wang, Y., Xu, N., Liu, A.-A., Li, W., & Zhang, Y. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3121062 · Cited by 95
LaTr: Layout-Aware Transformer for Scene-Text VQA
Biten, A. F., Litman, R., Xie, Y., Appalaraju, S., & Manmatha, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01605 · Cited by 97
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token ModelingPreprint
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12681 · Cited by 212
Self-citationFrom General to Specific: Informative Scene Graph Generation via Balance Adjustment
Guo, Y., Gao, L., Wang, X., Hu, Y., Xu, X., Lu, X., Shen, H. T., & Song, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01607 · Cited by 97
Adversarial attack and defense technologies in natural language processing: A survey
Qiu, S., Liu, Q., Zhou, S., & Huang, W. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.020 · Cited by 94
Adversarial Inference for Multi-Sentence Video Description
Park, J. S., Rohrbach, M., Darrell, T., & Rohrbach, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00676 · Cited by 94
Scene graph captioner: Image captioning based on structural visual representation
Xu, N., Liu, A.-A., Liu, J., Nie, W., & Su, Y. (2018) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2018.12.027 · Cited by 95
On Exploring Undetermined Relationships for Visual Relationship Detection
Zhan, Y., Yu, J., Yu, T., & Tao, D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00527 · Cited by 94
INGRESS: Interactive visual grounding of referring expressions
Shridhar, M., Mittal, D., & Hsu, D. (2020) · Int. J. Robotics Res. · doi:10.1177/0278364919897133 · Cited by 98
Object recognition datasets and challenges: A review
Salari, A., Djavadifar, A., Liu, X., & Najjaran, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.022 · Cited by 102
TubeDETR: Spatio-Temporal Video Grounding with Transformers
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01595 · Cited by 114
Text with Knowledge Graph Augmented Transformer for Video Captioning
Gu, X., Chen, G., Wang, Y., Zhang, L., Luo, T., & Wen, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01816 · Cited by 101
Gordo, A., & Larlus, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.560 · Cited by 93
Exploring Context and Visual Pattern of Relationship for Scene Graph Generation
Wang, W., Wang, R., Shan, S., & Chen, X. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00838 · Cited by 94
Learning to Predict Visual Attributes in the Wild
Pham, K., Kafle, K., Lin, Z., Ding, Z., Cohen, S. D., Tran, Q., & Shrivastava, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01282 · Cited by 99
Hierarchical Object-to-Zone Graph for Object Navigation
Zhang, S., Song, X., Bai, Y., Li, W., Chu, Y., & Jiang, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01485 · Cited by 98
Visually Grounded Reasoning across Languages and Cultures
Liu, F., Bugliarello, E., Ponti, E. M., Reddy, S., Collier, N., & Elliott, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.818 · Cited by 119
Zhang, L., Wang, J., Wang, Y., Sun, H. X., & Zhao, X. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104535 · Cited by 101
Q-Instruct: Improving Low-Level Visual Abilities for Multi-Modality Foundation Models
Wu, H., Zhang, Z., Zhang, E., Chen, C., Liao, L., Wang, A., Xu, K., Li, C., Hou, J., Zhai, G., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02408 · Cited by 130
“The primary problem is the lack of low-level visual datasets during training MLLMs, where publicly available datasets generally only focus on high-level visual abilities [2, 16, 22, 32].”
Hypergraph Attention Networks for Multimodal Learning
Kim, E., Kang, W. Y., On, K.-W., Heo, Y., & Zhang, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01459 · Cited by 98
Amplifying Key Cues for Human-Object-Interaction Detection
Liu, Y., Chen, Q., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_15 · Cited by 92
Contextual Heterogeneous Graph Network for Human-Object Interaction Detection
Wang, H., Zheng, W.-S., & Ling, Y. (2020) · European Conference on Computer Vision · doi:10.1007/978-3-030-58520-4_15 · Cited by 101
Stimuli-Aware Visual Emotion Analysis
Yang, J., Li, J., Wang, X., Ding, Y., & Gao, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3106813 · Cited by 100
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 152
“LLaVA-150K [49] COCO Caption [14] VQAv2 [26], GQA [34] OKVQA [55], A-OKVQA [71] OCRVQA [58], RefCOCO [36] Visual Genome [38] Video 2.5M WebVid-2.”
BackgroundMethodAlign2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
Datta, S., Sikka, K., Roy, A., Ahuja, K., Parikh, D., & Divakaran, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00269 · Cited by 90
Unsupervised Cross-Modal Hashing With Modality-Interaction
Tu, R.-C., Jiang, J., Lin, Q., Cai, C., Tian, S., Wang, H., & Liu, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3251395 · Cited by 94
“Genomes dataset [53] by [54], is used to extract the top m region proposals of xi .”
MethodGraph Representation Learning Meets Computer Vision: A Survey
Jiao, L., Chen, J., Liu, F., Yang, S., You, C., Liu, X., Li, L., & Hou, B. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3194869 · Cited by 95
Self-citationLanguage and Visual Entity Relationship Graph for Agent NavigationPreprint
Hong, Y., Rodríguez-Opazo, C., Qi, Y., Wu, Q., & Gould, S. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09304 · Cited by 99
Explicit Bias Discovery in Visual Question Answering Models
Manjunatha, V., Saini, N., & Davis, L. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00979 · Cited by 89
Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation
Hung, Z.-S., Mallya, A., & Lazebnik, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992222 · Cited by 92
Improving Image Captioning with Better Use of Caption
Shi, Z., Xu, Z., Qiu, X., & Zhu, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.664 · Cited by 89
Multimodal transformer networks for end-to-end video-grounded dialogue systems
LE, H., SAHOO, D., Chen, N. F., & Hoi, S. C. H. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 89
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning
Grunde-McLaughlin, M., Krishna, R., & Agrawala, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01113 · Cited by 89
Self-citationIncDet: In Defense of Elastic Weight Consolidation for Incremental Object Detection
Liu, L., Kuang, Z., Chen, Y., Xue, J.-H., Yang, W., & Zhang, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3002583 · Cited by 90
Learning Visual N-Grams from Web Data
Li, A., Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.449 · Cited by 94
Adaptive Cross-Modal Embeddings for Image-Text Alignment
Wehrmann, J., Kolling, C., & Barros, R. C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6915 · Cited by 88
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Sun, S., Chen, Y.-C., Li, L., Wang, S., Fang, Y., & Liu, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.77 · Cited by 94
Self-citationSpatial-Temporal Graphs for Cross-Modal Text2Video Retrieval
Song, X., Chen, J., Wu, Z., & Jiang, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3090595 · Cited by 88
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 195
The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
Wang, P., Wu, Q., Shen, C., & van den Hengel, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.416 · Cited by 90
Bailo, O., Ham, D., & Shin, Y. M. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00136 · Cited by 87
Fine-Grained Image Captioning With Global-Local Discriminative Objective
Wu, J., Chen, T., Wu, H., Yang, Z. Q., Luo, G., & Lin, L. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3011317 · Cited by 88
Class-Agnostic Object Detection with Multi-modal Transformer
Maaz, M., Rasheed, H., Khan, S. H., Khan, F. S., Anwer, R. M., & Yang, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_30 · Cited by 87
ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
Cheng, M., Sun, Y., Wang, L., Zhu, X., Yao, K., Chen, J., Song, G., Han, J., Liu, J., Ding, E., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00512 · Cited by 87
Controlling generative models with continuous factors of variationsPreprint
Plumerault, A., Borgne, H. L., & Hudelot, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.10238 · Cited by 105
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“Visual Genome (Krishna et al., 2017) connects structured image concepts to language and comprises 108,077 images along with detailed descriptions of all objects present in them, providing 5.4M region descriptions and 1.7M visual question-answer pairs.”
A survey on deep neural network-based image captioning
Liu, X., Xu, Q., & Wang, N. (2018) · The Visual Computer · doi:10.1007/s00371-018-1566-y · Cited by 89
Information Maximizing Visual Question Generation
Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00211 · Cited by 91
Self-citationBoosting Visual Question Answering with Context-aware Knowledge Aggregation
Li, G., Wang, X. Y., & Zhu, W. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413943 · Cited by 89
VD-BERT: A Unified Vision and Dialog Transformer with BERT
Wang, Y., Joty, S., Lyu, M. R.-T., King, I., Xiong, C., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.269 · Cited by 103
VisualMRC: Machine Reading Comprehension on Document Images
Tanaka, R., Nishida, K., & Yoshida, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17635 · Cited by 106
Unified Adaptive Relevance Distinguishable Attention Network for Image-Text Matching
Zhang, K., Mao, Z., Liu, A.-A., & Zhang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3141603 · Cited by 89
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
Gao, M., Chen, X., Niebles, J. C., Li, J., Xu, R., Liu, W., & Xiong, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_16 · Cited by 95
Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds
Jain, A., Gkanatsios, N., Mediratta, I., & Fragkiadaki, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_24 · Cited by 102
@ CREPE: Can Vision-Language Foundation Models Reason Compositionally?
Ma, Z., Hong, J., Gul, M. O., Gandhi, M., Gao, I., & Krishna, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01050 · Cited by 111
Self-citationLearning Canonical Representations for Scene Graph to Image Generation
Herzig, R., Bar, A., Xu, H., Chechik, G., Darrell, T., & Globerson, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_13 · Cited by 92
Dual-Glance Model for Deciphering Social Relationships
Li, J., Wong, Y., Zhao, Q., & Kankanhalli, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.289 · Cited by 85
Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning
Kim, D.-J., Choi, J., Oh, T.-H., & Kweon, I. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00643 · Cited by 86
Real-time Visual Object Tracking with Natural Language Description
Feng, Q., Ablavsky, V., Bai, Q., Li, G., & Sclaroff, S. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093425 · Cited by 83
Deep Convolutional Networks for Construction Object Detection Under Different Visual Conditions
Nath, N. D., & Behzadan, A. H. (2020) · Frontiers in Built Environment · doi:10.3389/fbuil.2020.00097 · Cited by 95
Context-aware Scene Graph Generation with Seq2Seq Transformers
Lu, Y., Rai, H., Chang, J. S., Knyazev, B. A., Yu, G., Shekhar, S., Taylor, G. W., & Volkovs, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01563 · Cited by 86
SST: Spatial and Semantic Transformers for Multi-Label Image Recognition
Chen, Z.-M., Cui, Q., Zhao, B., Song, R., Zhang, X., & Yoshie, O. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3148867 · Cited by 88
A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future
Zhu, C., & Chen, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3413013 · Cited by 113
“Models leveraging bipartite matching [13] and region-level grounding annotations [74], [75], [76] are also grouped into this methodology.”
BackgroundGPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Liu, Y., Chen, K., & Luo, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91813-1_4 · Cited by 323
Self-citationAnswer Them All! Toward Universal Visual Question Answering Models
Shrestha, R., Kafle, K., & Kanan, C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01072 · Cited by 87
Roses are Red, Violets are Blue… But Should VQA expect Them To?
Kervadec, C., Antipov, G., Baccouche, M., & Wolf, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00280 · Cited by 97
Learning Cross-Modal Context Graph for Visual Grounding
Liu, Y., Wan, B., Zhu, X., & He, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6833 · Cited by 85
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., & Kembhavi, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.707 · Cited by 99
On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
Wang, X., Liu, Y., Shen, C., Ng, C. C., Luo, C., Jin, L., Chan, C. S., van den Hengel, A., & Wang, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01014 · Cited by 90
Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder
Gouthaman, K. V., & Mittal, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_2 · Cited by 85
CSKG: The CommonSense Knowledge Graph
Ilievski, F., Szekely, P. A., & Zhang, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77385-4_41 · Cited by 92
Object-Centric Image Generation from Layouts
Sylvain, T., Zhang, P., Bengio, Y., Hjelm, R. D., & Sharma, S. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i3.16368 · Cited by 83
Zhu, X., Zhu, J., Li, H., Wu, X., Li, H., Wang, X., & Dai, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01630 · Cited by 108
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., & Wang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_30 · Cited by 105
Video Question Answering: Datasets, Algorithms and Challenges
Zhong, Y., Ji, W., Xiao, J., Li, Y., Deng, W., & Chua, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.432 · Cited by 133
TeCH: Text-Guided Reconstruction of Lifelike Clothed Humans
Huang, Y., Yi, H., Xiu, Y., Liao, T., Tang, J., Cai, D., & Thies, J. (2024) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv62453.2024.00152 · Cited by 116
“5* as T2I diffusion model, which is trained on LAION-5B, the VQA model BLIP [60] pretrained on 129M images from multiple datasets [14, 56, 69, 75, 78, 93] and fine-tuned on VQA2.”
MethodFooling Vision and Language Models Despite Localization and Attention Mechanism
Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., & Song, D. X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00520 · Cited by 92
NimbRo picking: Versatile part handling for warehouse automation
Schwarz, M., Milan, A., Lenz, C., Munoz, A., Periyasamy, A. S., Schreiber, M., Schuller, S., & Behnke, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989348 · Cited by 82
Visual Object Search by Learning Spatial Context
Druon, R., Yoshiyasu, Y., Kanezaki, A., & Watt, A. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2967677 · Cited by 82
Modality Shifting Attention Network for Multi-Modal Video Question Answering
Kim, J., Ma, M., Pham, T. X., Kim, K., & Yoo, C. D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01012 · Cited by 84
Metadata
- Authors
- Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- Technische Universität Dresden (DE)
- Yahoo (United States) (US)
- Snap (United States) (US)
- Centrum Wiskunde & Informatica (NL)
- Corresponding author
- Krishna, Ranjay
- ORCID
- Venue
- International Journal of Computer Vision
- Published
- 2017-02-06
- Volume
- 123
- Issue
- 1
- Pages
- 32-73
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2277195237
- Cited by
- 5,107
- References
- 123
- Citations by year
- 2026: 313 · 2025: 508 · 2024: 690 · 2023: 761 · 2022: 671 · 2021: 738 · 2020: 586 · 2019: 411 · 2018: 230 · 2017: 149
- FWCI
- 156.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Journal of Computer Vision (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands (submittedVersion)
- arXiv (Cornell University)
- Funders
- Brown Institute for Media Innovation
- Yahoo Inc.
- Stanford University Computer Science Department
- Office of Naval Research
- Toyota USA
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- object relations, picture description, question answering, object properties, WordNet synsets, visual cognition, crowdsourced annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
