The DOI matches its Crossref record; 118 of the 123 references it cites were found in a registry or an index.
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Open accessAbstract
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked “What vehicle is the person riding?”, computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) to answer correctly that “the person is riding a horse-drawn carriage.” In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 108K images where each image has an average of $$35$$ objects, $$26$$ attributes, and $$21$$ pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answer pairs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
Experimental — accuracy not yet measuredIn the citing paper
“For a few of these applications, some datasets were used like SQuAD, MS-MARCO and the like. Some works curated their own dataset owing to the distinctive application involved [122, 124].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
In the citing paper
“Also, most works used factoid-based questions but Yes/No [133], MCQ type [124] and reasoning type questions [122] were also generated for some applications.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [122]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in Ranjay Krishna Stanford University, Stanford, CA, USA E-mail: ranjaykrishna@cs.stanford.edu Yuke Zhu Stanford University, Stanford, CA, USA Oliver Groth Dresden University of…”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in Ranjay Krishna Stanford University, Stanford, CA, USA E-mail: ranjaykrishna@cs.stanford.edu Yuke Zhu Stanford University, Stanford, CA, USA Oliver Groth Dresden University of…”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“We then use the NeuralTalk (Karpathy and Fei-Fei, 2014) model to train a long short-term memory (LSTM) network (Hochreiter and Schmidhuber, 1997) to generate descriptions of regions.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 1
- Hayes, P. J. (1985). The second naive physics manifesto. Theories of the commonsense world (pp. 1–36).
Cited by
Sorted by citations
Items 401–500 of 1,032
Multi-Label Retinal Disease Classification Using Transformers
Rodríguez, M. A., Al-Marzouqi, H., & Liatsis, P. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3214086 · Cited by 137
DRAMA: Joint Risk Localization and Captioning in Driving
Malla, S., Choi, C., Dwivedi, I., Choi, J. H., & Li, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00110 · Cited by 128
Semantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 139
Sapiens: Foundation for Human Vision Models
Khirodkar, R., Bagautdinov, T., Martinez, J., Zhaoen, S., James, A., Selednik, P., Anderson, S., & Saito, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73235-5_12 · Cited by 213
MR Image Super-Resolution with Squeeze and Excitation Reasoning Attention Network
Zhang, Y., Li, K., Li, K., & Fu, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01322 · Cited by 129
CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation
Yu, J., Chai, Y., Wang, Y., Hu, Y., & Wu, Q. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/176 · Cited by 144
Jin, W., Cheng, Y., Shen, Y., Chen, W., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.197 · Cited by 174
Liu, F., Emerson, G., & Collier, N. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00566 · Cited by 212
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
Zhuge, M., Gao, D., Fan, D.-P., Jin, L., Chen, B., Zhou, H., Qiu, M., & Shao, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01246 · Cited by 130
Semantic Image Manipulation Using Scene Graphs
Dhamo, H., Farshad, A., Laina, I., Navab, N., Hager, G. D., Tombari, F., & Rupprecht, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00526 · Cited by 126
Scene-Aware Context Reasoning for Unsupervised Abnormal Event Detection in Videos
Sun, C., Jia, Y., Hu, Y., & Wu, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413887 · Cited by 123
Object Detection Meets Knowledge Graphs
Fang, Y., Kuan, K., Lin, J., Tan, C., & Chandrasekhar, V. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/230 · Cited by 122
Structured Attentions for Visual Question Answering
Zhu, C., Zhao, Y., Huang, S., Tu, K., & Ma, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.145 · Cited by 118
Detecting Unseen Visual Relations Using Analogies
Peyre, J., Šivic, J., Ivan Laptev, A., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00207 · Cited by 122
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual ConceptsPreprint
Zeng, Y., Zhang, X., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08276 · Cited by 219
Deep Learning for Image-to-Text Generation: A Technical Overview
He, X., & Deng, L. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2741510 · Cited by 126
Conditional Image-Text Embedding Networks
Plummer, B. A., Kordas, P., Kiapour, M. H., Zheng, S., Piramuthu, R., & Lazebnik, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_16 · Cited by 122
Varma, P., & Ré, C. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3291264.3291268 · Cited by 129
Hierarchical Modular Network for Video Captioning
Ye, H., Li, G., Qi, Y., Wang, S., Huang, Q., & Yang, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01741 · Cited by 125
Weakly-Supervised Visual Grounding of Phrases with Linguistic Structures
Xiao, F., Sigal, L., & Lee, Y. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.558 · Cited by 120
Learning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 128
Relationship Proposal Networks
Zhang, J., Elhoseiny, M., Cohen, S. D., Chang, W. H., & Elgammal, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.555 · Cited by 118
Detecting Human-Object Interactions with Action Co-occurrence Priors
Kim, D.-J., Sun, X. N., Choi, J., Lin, S. C.-F., & Kweon, I. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_43 · Cited by 119
Gan, C., Li, Y., Li, H., Sun, C., & Gong, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.201 · Cited by 116
Cross-Modal Relationship Inference for Grounding Referring Expressions
Yang, S., Li, G., & Yu, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00427 · Cited by 118
STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
Yoshikawa, Y., Shigeto, Y., & Takeuchi, A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2066 · Cited by 115
Detecting Human-Object Interactions via Functional Generalization
Bansal, A., Rambhatla, S. S., Shrivastava, A., & Chellappa, R. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6616 · Cited by 16
Federated Learning for Vision-and-Language Grounding Problems
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6824 · Cited by 115
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 130
Reasoning-RCNN: Unifying Adaptive Global Reasoning Into Large-Scale Object Detection
Xu, H., Jiang, C., Liang, X., Lin, L., & Li, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00658 · Cited by 114
Reflective Decoding Network for Image Captioning
Ke, L., Pei, W., Li, R., Shen, X., & Tai, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00898 · Cited by 115
Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset
Jia, M., Shi, M., Sirotenko, M., Cui, Y., Cardie, C., Hariharan, B., Adam, H., & Belongie, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_19 · Cited by 122
The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation
Li, L., Chen, L., Huang, Y., Zhang, Z., Zhang, S., & Xiao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01830 · Cited by 114
Self-citationScene Graph Generation: A comprehensive survey
Li, H., Zhu, G., Zhang, L., Jiang, Y., Dang, Y., Hou, H., Shen, P., Zhao, X., Shah, S. A. A., & Bennamoun, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127052 · Cited by 177
Query2Label: A Simple Transformer Way to Multi-Label ClassificationPreprint
Liu, S., Zhang, L., Yang, X., Su, H., & Zhu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10834 · Cited by 242
Learning from Collective Intelligence
Zhang, H., Shang, X., Luan, H., Wang, M., & Chua, T. (2016) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/2978656 · Cited by 114
Cross-modal knowledge reasoning for knowledge-based visual question answering
Yu, J., Zhu, Z., Wang, Y., Zhang, W., Hu, Y., & Tan, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107563 · Cited by 113
Fully Convolutional Scene Graph Generation
Liu, H., Yan, N., Mortazavi, M., & Bhanu, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01138 · Cited by 117
Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers
Feng, Q., Ablavsky, V., Bai, Q., & Sclaroff, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00579 · Cited by 127
Entity-Oriented Multi-Modal Alignment and Fusion Network for Fake News Detection
Li, P., Sun, X., Yu, H., Tian, Y., Yao, F., & Xu, G. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3098988 · Cited by 112
Physically Grounded Vision-Language Models for Robotic Manipulation
Gao, J., Sarkar, B., Xia, F., Xiao, T. G., Wu, J., Ichter, B., Majumdar, A., & Sadigh, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610090 · Cited by 185
Deep Adversarial Graph Attention Convolution Network for Text-Based Person Search
Liu, J., Zha, Z.-J., Hong, R., Wang, M., & Zhang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350991 · Cited by 112
Holistic 3D Scene Understanding from a Single Image with Implicit Representation
Zhang, C., Cui, Z., Zhang, Y., Zeng, B., Pollefeys, M., & Liu, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00872 · Cited by 120
Fine-Grained Scene Graph Generation with Data Transfer
Zhang, A., Yao, Y., Chen, Q., Ji, W., Liu, Z., Sun, M., & Chua, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_24 · Cited by 111
Causal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 141
“Visual Genome (https://homes.cs.washington.edu/ranjay/ visualgenome/api.html): The Visual Genome dataset [286], which comprises 108,000 images, is an invaluable resource for advancing cognitive development and improving image interpretation accuracy.”
BackgroundSelf-citationKim, U.-H., Park, J.-M., Song, T.-J., & Kim, J.-H. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2931042 · Cited by 109
PhraseCut: Language-Based Image Segmentation in the Wild
Wu, C., Lin, Z., Cohen, S. D., Bui, T., & Maji, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01023 · Cited by 124
Recovering the Unbiased Scene Graphs from the Biased Ones
Chiou, M.-J., Ding, H., Yan, H., Wang, C., Zimmermann, R., & Feng, J. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475297 · Cited by 113
MVImgNet: A Large-scale Dataset of Multi-view Images
Yu, X., Xu, M., Zhang, Y., Liu, H., Ye, C., Wu, Y., Yan, Z., Zhu, C., Xiong, Z., Liang, T., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00883 · Cited by 139
“Some other datasets include PASCAL VOC [28], Visual Genome [53], Cityscapes [22], MPII [4], etc.”
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 165
“Such uncurated incidentally created data is often lower-quality and noisier than purpose-created and curated datasets such as COCO [20] and Visual Genome [26].”
Recent Advances in Object Detection in the Age of Deep Convolutional Neural NetworksPreprint
Agarwal, S., du Terrail, J. O., & Jurie, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03193 · Cited by 145
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
Gan, Z., Cheng, Y., Kholy, A. E., Li, L., Liu, J., & Gao, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1648 · Cited by 112
Self-citationHandwritten Mathematical Expression Recognition via Paired Adversarial Learning
Wu, J.-W., Yin, F., Zhang, Y.-M., Zhang, X.-Y., & Liu, C.-L. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01291-5 · Cited by 105
Richpedia: A Large-Scale, Comprehensive Multi-Modal Knowledge Graph
Wang, M., Wang, H., Qi, G., & Zheng, Q. (2020) · Big Data Research · doi:10.1016/j.bdr.2020.100159 · Cited by 109
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 118
Dual Global Enhanced Transformer for image captioning
Xian, T., Li, Z., Zhang, C., & Ma, H. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.01.011 · Cited by 113
Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00419 · Cited by 108
Large-Scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
Murahari, V., Batra, D., Parikh, D., & Das, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_20 · Cited by 109
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip RetrievalPreprint
Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., & Li, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08860 · Cited by 108
Commonsense Knowledge in Machine Intelligence
Tandon, N., Varde, A. S., & de Melo, G. (2018) · ACM SIGMOD Record · doi:10.1145/3186549.3186562 · Cited by 107
Aligning Linguistic Words and Visual Semantic Units for Image Captioning
Guo, L., Liu, J., Tang, J., Li, J., Luo, W., & Lu, H. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350943 · Cited by 107
Gated attention fusion network for multimodal sentiment classification
Du, Y., Liu, Y., Peng, Z. J., & Jin, X. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.108107 · Cited by 107
Cross on Cross Attention: Deep Fusion Transformer for Image Captioning
Zhang, J., Xie, Y., Ding, W., & Wang, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3243725 · Cited by 109
“Inspired by the human visual system, Anderson et al. [24] first use Faster R-CNN [25] pre-trained on the Visual Genome dataset [26] to extract a series of region features from images, which can provide object-level information.”
MethodOptimization of Image Transmission in Cooperative Semantic Communication Networks
Zhang, W., Wang, Y., Chen, M., Luo, T., & Niyato, D. T. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2023.3282906 · Cited by 115
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene GraphPreprint
Yu, F. R., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16934 · Cited by 106
Towards Unsupervised Image Captioning With Shared Multimodal Embeddings
Laina, I., Rupprecht, C., & Navab, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00751 · Cited by 111
Focal Visual-Text Attention for Visual Question Answering
Liang, J., Jiang, L., Cao, L., Li, L.-J., & Hauptmann, A. G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00642 · Cited by 103
Self-citationContext-Aware Visual Policy Network for Sequence-Level Image Captioning
Liu, D., Zha, Z.-J., Zhang, H., Zhang, Y., & Wu, F. H. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240632 · Cited by 110
Bugliarello, E., Cotterell, R., Okazaki, N., & Elliott, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00408 · Cited by 133
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 273
“OurLVLM-eHubconsistsof13representative Index Terms —Vision-language model, multimodal evaluation, large language model, multi-turn evaluation.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering
Narasimhan, M., & Schwing, A. G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_28 · Cited by 107
MRA-Net: Improving VQA Via Multi-Modal Relation Attention Network
Peng, L., Yang, Y., Wang, Z., Huang, Z. H., & Shen, H. T. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3004830 · Cited by 103
Symbiotic Attention for Egocentric Action Recognition With Object-Centric Alignment
Wang, X., Zhu, L., Wu, Y., & Yang, Y. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3015894 · Cited by 106
FlexiViT: One Model for All Patch Sizes
Beyer, L., Izmailov, P., Kolesnikov, A., Caron, M., Kornblith, S., Zhai, X., Minderer, M., Tschannen, M., Alabdulmohsin, I., & Pavetic, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01393 · Cited by 132
Compositional Chain-of-Thought Prompting for Large Multimodal Models
Mitra, C., Huang, B., Darrell, T., & Herzig, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01367 · Cited by 150
“1 For example, Visual Genome [37] contains only ∼ 100 K image-SG pairs, which is smaller than the existing LMMs pretraining datasets.”
BackgroundAlpha-CLIP: A CLIP Model Focusing on Wherever you Want
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01237 · Cited by 151
“5 [29] on Visual Genome [20] and Ref-COCOg [31].”
Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences
Fries, J., Varma, P., Chen, V. S., Xiao, K., Tejeda, H., Saha, P., Dunnmon, J., Chubb, H., Maskatia, S. A., Fiterau, M., et al. (2019) · Nature Communications · doi:10.1038/s41467-019-11012-3 · Cited by 105
Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences
Zhang, Z., Zhao, Z., Zhao, Y., Wang, Q., Liu, H., & Gao, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01068 · Cited by 104
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.-C., & Liu, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_34 · Cited by 112
SAT: 2D Semantics Assisted Training for 3D Visual Grounding
Yang, Z., Zhang, S., Wang, L., & Luo, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00187 · Cited by 111
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 126
Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
Wang, T., Li, F., Zhu, L., Li, J., Zhang, Z., & Shen, H. T. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3525147 · Cited by 161
“COCO [226], NUS-WIDE [227], IAPR TC-12 [228], Xmedia [229], Fashion-Gen [230], Recipe1M [231], CUB 200-2011 [232], Caltech-256 [233], LabelMe [234], Visual Genome [235], CC3M [236].”
Graph-Structured Referring Expression Reasoning in the Wild
Yang, S., Li, G., & Yu, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00997 · Cited by 100
Deep Multimodal Neural Architecture Search
Zhou, Y., Cui, Y., Yu, J., Wang, M., Tao, D., & Tian, Q. C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413977 · Cited by 106
Li, Y., Chen, R., Zhang, Y., Zhang, M., & Chen, L. (2020) · Remote Sensing · doi:10.3390/rs12234003 · Cited by 102
See, Hear, and Read: Deep Aligned RepresentationsPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00932 · Cited by 135
The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
Iyyer, M., Manjunatha, V., Guha, A., Vyas, Y., Boyd-Graber, J. L., Daumé, H., & Davis, L. S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.686 · Cited by 107
Learning to Collocate Neural Modules for Image Captioning
Yang, X., Zhang, H., & Cai, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00435 · Cited by 100
Learning of Visual Relations: The Devil is in the Tails
Desai, A., Wu, T.-Y., Tripathi, S., & Vasconcelos, N. M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01512 · Cited by 102
FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback
Goenka, S., Zheng, Z., Jaiswal, A., Chada, R., Wu, Y., Hedau, V., & Natarajan, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01371 · Cited by 109
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100
Sharma, D., Purushotham, S., & Reddy, C. K. A. (2021) · Scientific Reports · doi:10.1038/s41598-021-98390-1 · Cited by 106
MixGen: A New Multi-Modal Data Augmentation
Hao, X., Zhu, Y., Appalaraju, S., Zhang, A., Zhang, W., Li, B., & Li, M. (2023) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw58289.2023.00042 · Cited by 110
Self-citationDetecting Everything in the Open World: Towards Universal Object Detection
Wang, Z., Li, Y., Chen, X., Lim, S.-N., Torralba, A. B., Zhao, H., & Wang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01100 · Cited by 111
“We mainly perform inference on the LVIS [17], Ima-geNetBoxes [24], and VisualGenome [23] datasets to evaluate the detector’s open-world performance.”
InfluentialGrounded Situation Recognition
Pratt, S. M., Yatskar, M., Weihs, L., Farhadi, A., & Kembhavi, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_19 · Cited by 108
Re-Attention for Visual Question Answering
Guo, W., Zhang, Y., Yang, J., & Yuan, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3097180 · Cited by 98
Video Graph Transformer for Video Question Answering
Xiao, J., Zhou, P., Chua, T., & Yan, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_3 · Cited by 108
Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
Kuo, C.-W., & Kira, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01744 · Cited by 99
Learning to Segment via Cut-and-Paste
Remez, T., Huang, J., & Brown, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_3 · Cited by 98
Metadata
- Authors
- Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- Technische Universität Dresden (DE)
- Yahoo (United States) (US)
- Snap (United States) (US)
- Centrum Wiskunde & Informatica (NL)
- Corresponding author
- Krishna, Ranjay
- ORCID
- Venue
- International Journal of Computer Vision
- Published
- 2017-02-06
- Volume
- 123
- Issue
- 1
- Pages
- 32-73
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2277195237
- Cited by
- 5,107
- References
- 123
- Citations by year
- 2026: 313 · 2025: 508 · 2024: 690 · 2023: 761 · 2022: 671 · 2021: 738 · 2020: 586 · 2019: 411 · 2018: 230 · 2017: 149
- FWCI
- 156.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Journal of Computer Vision (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands (submittedVersion)
- arXiv (Cornell University)
- Funders
- Brown Institute for Media Innovation
- Yahoo Inc.
- Stanford University Computer Science Department
- Office of Naval Research
- Toyota USA
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- object relations, picture description, question answering, object properties, WordNet synsets, visual cognition, crowdsourced annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
