The DOI matches its Crossref record; 118 of the 123 references it cites were found in a registry or an index.
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Open accessAbstract
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked “What vehicle is the person riding?”, computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) to answer correctly that “the person is riding a horse-drawn carriage.” In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 108K images where each image has an average of $$35$$ objects, $$26$$ attributes, and $$21$$ pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answer pairs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For a few of these applications, some datasets were used like SQuAD, MS-MARCO and the like. Some works curated their own dataset owing to the distinctive application involved [122, 124].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
In the citing paper
“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
In the citing paper
“Also, most works used factoid-based questions but Yes/No [133], MCQ type [124] and reasoning type questions [122] were also generated for some applications.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in Ranjay Krishna Stanford University, Stanford, CA, USA E-mail: ranjaykrishna@cs.stanford.edu Yuke Zhu Stanford University, Stanford, CA, USA Oliver Groth Dresden University of…”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in Ranjay Krishna Stanford University, Stanford, CA, USA E-mail: ranjaykrishna@cs.stanford.edu Yuke Zhu Stanford University, Stanford, CA, USA Oliver Groth Dresden University of…”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“We then use the NeuralTalk (Karpathy and Fei-Fei, 2014) model to train a long short-term memory (LSTM) network (Hochreiter and Schmidhuber, 1997) to generate descriptions of regions.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 1
- Hayes, P. J. (1985). The second naive physics manifesto. Theories of the commonsense world (pp. 1–36).
Cited by
Sorted by citations
Items 201–300 of 1,032
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A SurveyPreprint
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06796 · Cited by 253
An Analysis of Visual Question Answering Algorithms
Kafle, K., & Kanan, C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.217 · Cited by 268
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Yu, L., Tan, H. X., Bansal, M., & Berg, T. L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.375 · Cited by 250
Transferable Interactiveness Knowledge for Human-Object Interaction Detection
Li, Y., Zhou, S., Huang, X., Xu, L., Ma, Z., Fang, H.-S., Wang, Y., & Lu, C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00370 · Cited by 255
Pose-Aware Multi-Level Feature Network for Human Object Interaction Detection
Wan, B., Zhou, D., Liu, Y., Li, R., & He, X. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00956 · Cited by 250
ShareGPT4V: Improving Large Multi-modal Models with Better Captions
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., & Lin, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72643-9_22 · Cited by 816
Hierarchical LSTMs with Adaptive Attention for Visual Captioning
Gao, L., Li, X., Song, J., & Shen, H. T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894139 · Cited by 249
Deep Variation-Structured Reinforcement Learning for Visual Relationship and Attribute Detection
Liang, X., Lee, L., & Xing, E. P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.469 · Cited by 252
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation
Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264
Visual question answering: Datasets, algorithms, and future challenges
Kafle, K., & Kanan, C. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.005 · Cited by 259
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions
Wald, J., Dhamo, H., Navab, N., & Tombari, F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00402 · Cited by 246
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
Li, W., Gao, C., Niu, G., Xiao, X., Liu, H., Liu, J., Wu, H. R., & Wang, H. (2021) · doi:10.18653/v1/2021.acl-long.202 · Cited by 362
RelTR: Relation Transformer for Scene Graph Generation
Cong, Y., Yang, M. Y., & Rosenhahn, B. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3268066 · Cited by 272
Recognize Anything: A Strong Image Tagging Model
Zhang, Y., Huang, X., Ma, J., Li, Z., Luo, Z., Xie, Y., Qin, Y., Luo, T., Li, Y., Liu, S., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00179 · Cited by 242
“The 4M setting includes two human-annotated datasets, COCO [16] (113K images, 557K captions) and Visual Genome [13] (101K images, 822K captions), along with two large-scale web-based datasets, Conceptual Cap-tions [6] (3M images, 3M captions) and SBU Cap-tions [21] (849K images, 849K captions).”
Explicit Knowledge-based Reasoning for Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/179 · Cited by 296
Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification
Xu, A., Yao, A., Li, A., Liang, A., & Zhang, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00675 · Cited by 228
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 261
Iterative Visual Reasoning Beyond Convolutions
Chen, X., Li, L.-J., Fei-Fei, L., & Gupta, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00756 · Cited by 236
Self-citationViP-CNN: Visual Phrase Guided Convolutional Neural Network
Li, Y., Ouyang, W., Wang, X., & Tang, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.766 · Cited by 237
A Real-Time Cross-Modality Correlation Filtering Method for Referring Expression Comprehension
Liao, Y., Liu, S., Li, G., Wang, F., Chen, Y., Qian, C., & Li, B. H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01089 · Cited by 242
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2018) · arXiv · doi:10.48550/arxiv.1809.02165 · Cited by 2,965
A Survey of Single-Scene Video Anomaly Detection
Ramachandra, B., Jones, M., & Vatsavai, R. R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3040591 · Cited by 259
Image retrieval from remote sensing big data: A survey
Li, Y., Ma, J., & Zhang, Y. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.10.008 · Cited by 244
Multi-level Attention Networks for Visual Question Answering
Yu, D., Fu, J., Mei, T., & Rui, Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.446 · Cited by 236
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 296
BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection
Ben-younes, H., Cadène, R., Thome, N., & Cord, M. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33018102 · Cited by 235
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
Huang, Z., Zeng, Z., Huang, Y., Liu, B., Fu, D., & Fu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01278 · Cited by 238
Visual Semantic Navigation using Scene PriorsPreprint
Yang, W. P., Wang, X., Farhadi, A., Gupta, A., & Mottaghi, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06543 · Cited by 229
Revisiting Visual Question Answering Baselines
Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_44 · Cited by 238
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA
Hu, R., Singh, A., Darrell, T., & Rohrbach, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01001 · Cited by 234
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations through Scene Graphs
Yu, F. R., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16431 · Cited by 358
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 272
Prompting Large Language Models with Answer Heuristics for Knowledge-Based Visual Question Answering
Shao, Z., Zhou, Y., Wang, M., & Yu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01438 · Cited by 240
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 232
Hierarchy Parsing for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00271 · Cited by 219
Negative-Aware Attention Framework for Image-Text Matching
Zhang, K., Mao, Z., Wang, Q., & Zhang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01521 · Cited by 216
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
Cornia, M., Baraldi, L., & Cucchiara, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00850 · Cited by 213
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 239
Zhao, B., Meng, L., Yin, W., & Sigal, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00878 · Cited by 218
Bridging Knowledge Graphs to Generate Scene Graphs
Zareian, A., Karaman, S., & Chang, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58592-1_36 · Cited by 219
LXMERT: Learning Cross-Modality Encoder Representations from TransformersPreprint
Tan, H. X., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07490 · Cited by 729
Recurrent Topic-Transition GAN for Visual Paragraph Generation
Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.364 · Cited by 209
Adversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 228
Linear Spectral Clustering Superpixel
Chen, J., Li, Z., & Huang, B. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2651389 · Cited by 207
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans
Rosinol, A., Gupta, A., Abate, M., Shi, J., & Carlone, L. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.079 · Cited by 266
Multilabel Image Classification With Regional Latent Semantic Dependencies
Zhang, J., Wu, Q., Shen, C., Zhang, J., & Lu, J. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2812605 · Cited by 206
GRES: Generalized Referring Expression Segmentation
Liu, C., Ding, H., & Jiang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02259 · Cited by 241
“Image Source CLEF [8] COCO [25] VG [22] COCO [25]”
BackgroundKRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Marino, K., Chen, X., Parikh, D., Gupta, A., & Rohrbach, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01389 · Cited by 211
A Review on Methods and Applications in Multimodal Deep Learning
Jabeen, S., Li, X., Amin, M. S., Bourahla, O. E. F., Li, S., & Jabbar, A. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3545572 · Cited by 225
A Survey of Vision-Language Pre-Trained Models
Du, Y., Liu, Z., Li, J., & Zhao, W. X. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/762 · Cited by 313
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 438
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., & Batra, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_16 · Cited by 201
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 219
PMC-CLIP: Contrastive Language-Image Pre-training Using Biomedical Documents
Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., & Xie, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43993-3_51 · Cited by 249
Wang, K., Lin, Y.-A., Weissmann, B., Savva, M., Chang, A. X., & Ritchie, D. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322941 · Cited by 200
Shamrat, F. M. J. M., Azam, S., Karim, A., Ahmed, K., Bui, F. M., & Boer, F. D. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106646 · Cited by 215
Generalized Decoding for Pixel, Image, and Language
Zou, X., Dou, Z.-Y., Yang, J., Gan, Z., Li, L., Li, C., Dai, X., Behl, H. S., Wang, J., Yuan, L., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01451 · Cited by 242
Self-citationGLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E. P., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01236 · Cited by 302
Discriminability Objective for Training Descriptive Captions
Luo, R., Shakhnarovich, G., Cohen, S. D., & Price, B. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00728 · Cited by 194
Unpaired Image Captioning via Scene Graph Alignments
Gu, J., Joty, S., Cai, J., Zhao, H., Yang, X., & Wang, G. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01042 · Cited by 193
Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition
Chen, T., Lin, L., Chen, R., Hui, X., & Wu, H. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3025814 · Cited by 196
Dynamic Multimodal Instance Segmentation Guided by Natural Language Queries
Margffoy-Tuay, E. A., Pérez, J. C., Botero, E., & Arbeláez, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_39 · Cited by 191
Ji, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16258 · Cited by 211
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00914 · Cited by 189
Zhu, Y., Tian, Y., Metaxas, D., & Dollár, P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.320 · Cited by 205
Self-citationZhou, L., Kalantidis, Y., Chen, X., Corso, J. J., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00674 · Cited by 191
Self-citationMathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., & Jawahar, C. V. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00264 · Cited by 326
Know More Say Less: Image Captioning Based on Scene Graphs
Li, X., & Jiang, S. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896516 · Cited by 181
Kang, J., Du, H., Li, Z., Xiong, Z., Ma, S., Niyato, D. T., & Li, Y. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221990 · Cited by 203
LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation
Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y., & Li, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02154 · Cited by 229
“Experiments in challenging COCO-stuff and Visual Genome (VG) show that our proposed method significantly outperforms both stateof-the-art GAN-based and diffusion-based methods in various evaluation metrics.”
MethodResultInfluentialmPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., & Huang, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01239 · Cited by 431
“∈ R d × d learnable projection matrices, and C l ∈ R ( L V + L T ) × d context features of l -th layer.”
BackgroundVideoChat: chat-centric video understanding
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, L., Wang, L., & Qiao, Y. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4321-9 · Cited by 1,181
“Compared with detailed video descriptions, video conversations are introduced to further improve the diversity, temporal and casual features in the video instruction data.”
Specifying Object Attributes and Relations in Interactive Scene Generation
Ashual, O., & Wolf, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00466 · Cited by 185
TVQA+: Spatio-Temporal Grounding for Video Question Answering
Lei, J., Yu, L., Berg, T. L., & Bansal, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.730 · Cited by 225
Medical visual question answering: A survey
Lin, Z., Zhang, D., Tao, Q., Shi, D., Haffari, G., Wu, Q., He, M., & Ge, Z. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102611 · Cited by 233
The Devil Is in Classification: A Simple Framework for Long-Tail Instance Segmentation
Wang, T., Yu, L. Z., Kang, B., Li, J., Liew, J. H., Tang, S., Hoi, S. C. H., & Feng, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_43 · Cited by 185
Weakly-Supervised Learning of Visual Relations
Peyre, J., Ivan Laptev, A., Schmid, C., & Šivic, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.554 · Cited by 182
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182
Video Object Segmentation with Language Referring Expressions
Khoreva, A., Rohrbach, A., & Schiele, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_8 · Cited by 193
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.488 · Cited by 282
RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
Zhan, Y., Xiong, Z., & Yuan, Y. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3250471 · Cited by 179
“Over the years, various visual grounding datasets [18], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65] based on real-world and computer-generated images have been proposed to study visual grounding.”
BackgroundMethodAlign and Prompt: Video-and-Language Pre-training with Entity Prompts
Li, D., Li, J., Li, H., Niebles, J. C., & Hoi, S. C. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00490 · Cited by 187
Connecting Vision and Language with Localized Narratives
Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., & Ferrari, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_38 · Cited by 188
CPT: Colorful Prompt Tuning for pre-trained vision-language models
Yao, Y., Zhang, A., Zhang, Z., Liu, Z., Chua, T., & Sun, M. (2024) · AI Open · doi:10.1016/j.aiopen.2024.01.004 · Cited by 262
Causal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175
Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
Hatori, J., Kikuchi, Y., Kobayashi, S., Takahashi, K., Tsuboi, Y., Unno, Y., Ko, W., & Tan, J. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460699 · Cited by 174
Dense Captioning with Joint Inference and Visual Context
Yang, L., Tang, K., Yang, J., & Li, L.-J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.214 · Cited by 174
Self-citationSee Finer, See More: Implicit Modality Alignment for Text-Based Person Retrieval
Shu, X., Wen, W. X., Wu, H., Chen, K., Song, Y., Qiao, R., Ren, B., & Wang, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25072-9_42 · Cited by 181
Hendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171
KVQA: Knowledge-Aware Visual Question Answering
Shah, S. B., Mishra, A., Yadati, N., & Talukdar, P. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018876 · Cited by 185
On the Origin of Deep LearningPreprint
Wang, H., & Raj, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07800 · Cited by 312
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 571
“Multi-Instruct covers most multi-modal tasks that require visual understanding and multi-modal reasoning, such as Visual Question Answering [11, 41, 30], Image Captioning [16], Image Generation [7], and Visual Relationship Understanding [14], among others.”
BackgroundSpatial Memory for Context Reasoning in Object Detection
Chen, X., & Gupta, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.440 · Cited by 175
Few-Example Object Detection with Model Communication
Dong, X., Zheng, L., Ma, F., Yang, Y., & Meng, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2844853 · Cited by 175
End-to-End Transformer Based Model for Image Captioning
Wang, Y., Xu, J., & Sun, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20160 · Cited by 183
A Survey on Long-Tailed Visual Recognition
Yang, L., Jiang, H., Song, Q., & Guo, J. H. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01622-8 · Cited by 179
Counterfactual Critic Multi-Agent Training for Scene Graph Generation
Chen, L., Zhang, H., Xiao, J., He, X., Pu, S., & Chang, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00471 · Cited by 166
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W., Zheng, Y., & Shou, M. Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00685 · Cited by 218
“Datasets: Current layout-to-image methods are mainly trained on paired layout-image data of COCO-Stuff [5] or VG [17].”
Method
Metadata
- Authors
- Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- Technische Universität Dresden (DE)
- Yahoo (United States) (US)
- Snap (United States) (US)
- Centrum Wiskunde & Informatica (NL)
- Corresponding author
- Krishna, Ranjay
- ORCID
- Venue
- International Journal of Computer Vision
- Published
- 2017-02-06
- Volume
- 123
- Issue
- 1
- Pages
- 32-73
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2277195237
- Cited by
- 5,107
- References
- 123
- Citations by year
- 2026: 313 · 2025: 508 · 2024: 690 · 2023: 761 · 2022: 671 · 2021: 738 · 2020: 586 · 2019: 411 · 2018: 230 · 2017: 149
- FWCI
- 156.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Journal of Computer Vision (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands (submittedVersion)
- arXiv (Cornell University)
- Funders
- Brown Institute for Media Innovation
- Yahoo Inc.
- Stanford University Computer Science Department
- Office of Naval Research
- Toyota USA
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- object relations, picture description, question answering, object properties, WordNet synsets, visual cognition, crowdsourced annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
