Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 701–800 of 1,041
Improving Image Captioning with Better Use of Caption
Shi, Z., Xu, Z., Qiu, X., & Zhu, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.664 · Cited by 102
Text-Only Training for Image Captioning using Noise-Injected CLIP
Nukrai, D., Mokady, R., & Globerson, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.299 · Cited by 85
Learning Visual N-Grams from Web Data
Li, A., Jabri, A., Joulin, A., & van der Maaten, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.449 · Cited by 84
Sitcom-star-based clothing retrieval for video advertising: a deep learning framework
Zhang, H., Ji, Y., Huang, W., & Liu, L. (2018) · Neural Computing and Applications · doi:10.1007/s00521-018-3579-x · Cited by 92
Lu, P., Li, H., Zhang, W., Wang, J., & Wang, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12240 · Cited by 96
Efficient processing of GRU based on word embedding for text classification
Zulqarnain, M., Ghazali, R., Ghouse, M. G., & Mushtaq, M. F. (2019) · JOIV International Journal on Informatics Visualization · doi:10.30630/joiv.3.4.289 · Cited by 113
Adaptive Cross-Modal Embeddings for Image-Text Alignment
Wehrmann, J., Kolling, C., & Barros, R. C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6915 · Cited by 90
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Sun, S., Chen, Y.-C., Li, L., Wang, S., Fang, Y., & Liu, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.77 · Cited by 111
Chen, W., Luo, J., Zhang, F., & Tian, Z. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17949-4 · Cited by 120
Book2Movie: Aligning video scenes with book chapters
Tapaswi, M., Bauml, M., & Stiefelhagen, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298792 · Cited by 87
“Factual” or “Emotional”: Stylized Image Captioning with Adaptive Learning and Attention
Chen, T., Zhang, Z., You, Q., Chen, F., Wang, Z., Jin, H., & Luo, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_32 · Cited by 107
A Visual Attention Grounding Neural Model for Multimodal Machine Translation
Zhou, M., Cheng, R., Lee, Y. J., & Yu, Z. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1400 · Cited by 83
Fine-Grained Image Captioning With Global-Local Discriminative Objective
Wu, J., Chen, T., Wu, H., Yang, Z. Q., Luo, G., & Lin, L. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3011317 · Cited by 94
Dual Compositional Learning in Interactive Image Retrieval
Kim, J.-S., Yu, Y., Kim, H., & Kim, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16271 · Cited by 92
To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review
Shwartz-Ziv, R., & LeCun, Y. (2024) · Entropy · doi:10.3390/e26030252 · Cited by 195
A survey on deep neural network-based image captioning
Liu, X., Xu, Q., & Wang, N. (2018) · The Visual Computer · doi:10.1007/s00371-018-1566-y · Cited by 95
Recognizing American Sign Language Gestures from Within Continuous Videos
Ye, Y., Tian, Y., Huenerfauth, M., & Liu, J. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00280 · Cited by 100
Unified Adaptive Relevance Distinguishable Attention Network for Image-Text Matching
Zhang, K., Mao, Z., Liu, A.-A., & Zhang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3141603 · Cited by 97
Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds
Jain, A., Gkanatsios, N., Mediratta, I., & Fragkiadaki, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_24 · Cited by 183
Salisbury, E., Kamar, E., & Morris, M. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13301 · Cited by 83
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
Huang, D.-A., Buch, S., Dery, L. M., Garg, A., Fei-Fei, L., & Niebles, J. C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00623 · Cited by 106
Self-citationGroupCap: Group-Based Image Captioning with Structured Relevance and Diversity Constraints
Chen, F., Ji, R., Sun, X., Wu, Y., & Su, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00146 · Cited by 86
Asymmetric GAN for Unpaired Image-to-Image Translation
Li, Y., Tang, S., Zhang, R., Zhang, Y., Li, J., & Yan, S. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2922854 · Cited by 97
Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning
Kim, D.-J., Choi, J., Oh, T.-H., & Kweon, I. S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00643 · Cited by 101
Bearing Fault Diagnosis Method Based on EEMD and LSTM
Zou, P., Hou, B., Jiang, L., & Zhang, Z. (2020) · International Journal of Computers Communications & Control · doi:10.15837/ijccc.2020.1.3780 · Cited by 86
Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning
Salvador, A., Gündoğdu, E., Bazzani, L., & Donoser, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01522 · Cited by 88
Leveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection
Singhal, S., Pandey, T., Mrig, S., Shah, R. R., & Kumaraguru, P. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524650 · Cited by 127
Explicit Sparse Transformer: Concentrated Attention Through Explicit SelectionPreprint
Zhao, G., Lin, J., Zhang, Z., Ren, X., Su, Q., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11637 · Cited by 191
Leveraging Visual Question Answering for Image-Caption Ranking
Lin, X., & Parikh, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_17 · Cited by 89
Big Healthcare Data Analytics: Challenges and Applications
Lee, C., Luo, Z., Ngiam, K. Y., Zhang, M., Zheng, K., Chen, G., Ooi, B. C., & Yip, W. L. J. (2017) · Scalable computing and communications · doi:10.1007/978-3-319-58280-1_2 · Cited by 118
Semi-supervised robust deep neural networks for multi-label image classification
Çevıkalp, H., Benlıgıray, B., & Gerek, Ö. N. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107164 · Cited by 88
PoPPL: Pedestrian Trajectory Prediction by LSTM With Automatic Route Class Clustering
Xue, H., Huynh, D. Q., & Reynolds, M. A. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2975837 · Cited by 93
Re-Caption: Saliency-Enhanced Image Captioning Through Two-Phase Learning
Zhou, L., Zhang, Y., Jiang, Y., Zhang, T., & Fan, W. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2928144 · Cited by 88
Insertion-based Decoding with Automatically Inferred Generation Order
Gu, J., Liu, Q., & Cho, K. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00292 · Cited by 84
Radical analysis network for learning hierarchies of Chinese characters
Zhang, J., Du, J., & Dai, L.-R. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107305 · Cited by 82
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling
Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., & Wang, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_30 · Cited by 183
Adding Chinese Captions to Images
Li, X., Lan, W., Dong, J., & Liu, H. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912049 · Cited by 105
Fooling Vision and Language Models Despite Localization and Attention Mechanism
Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., & Song, D. X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00520 · Cited by 111
Transferable Representation Learning in Vision-and-Language Navigation
Huang, H., Jain, V., Mehta, H., Ku, A., Magalhaes, G., Baldridge, J., & Ie, E. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00750 · Cited by 82
Alternating Co-Quantization for Cross-Modal Hashing
Irie, G., Arai, H., & Taniguchi, Y. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.219 · Cited by 86
Image Caption Generation Using Deep Learning Technique
Amritkar, C., & Jabade, V. (2018) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea.2018.8697360 · Cited by 82
Regularized Two-Branch Proposal Networks for Weakly-Supervised Moment Retrieval in Videos
Zhang, Z., Lin, Z., Zhao, Z., Zhu, J., & He, X. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413967 · Cited by 93
Incorporating attentive multi-scale context information for image captioning
Prudviraj, J., Yenduri, S., & Mohan, C. K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11895-9 · Cited by 85
GrapHD: Graph-Based Hyperdimensional Memorization for Brain-Like Cognitive Learning
Poduval, P., Alimohamadi, H., Zakeri, A., Imani, F., Najafi, M. H., Givargis, T., & Imani, M. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.757125 · Cited by 112
Representation Learning for Natural Language Processing
Liu, Z., Lin, Y., & Sun, M. (2020) · Springer eBooks · doi:10.1007/978-981-15-5573-2 · Cited by 140
Kumari, S., & Singh, P. P. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107912 · Cited by 118
Perception science in the age of deep neural networks.
VanRullen, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00142 · Cited by 82
User-guided Hierarchical Attention Network for Multi-modal Social Image Popularity Prediction
Zhang, W., Wang, W., Wang, J., & Zha, H. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186026 · Cited by 99
VisualWord2Vec (Vis-W2V): Learning Visually Grounded Word Embeddings Using Abstract Scenes
Kottur, S., Vedantam, R., Moura, J. M. F., & Parikh, D. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.539 · Cited by 102
Attentive Linear Transformation for Image Captioning
Ye, S., Han, J., & Liu, N. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2855406 · Cited by 82
Information fusion in visual question answering: A Survey
Zhang, D., Cao, R., & Wu, S. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.03.005 · Cited by 87
X -Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning
Yuan, Z., Yan, X., Liao, Y., Guo, Y., Li, G., Cui, S. R., & Li, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00837 · Cited by 127
Paying Attention to Descriptions Generated by Image Captioning Models
Tavakoliy, H. R., Shetty, R., Borji, A., & Laaksonen, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.272 · Cited by 81
Learning to Guide Decoding for Image Captioning
Jiang, W., Ma, L., Chen, X., Zhang, H., & Liu, W. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12283 · Cited by 78
Zero-Shot Object Detection with Textual Descriptions
Li, Z., Yao, L., Zhang, X., Wang, X., Kanhere, S. S., & Zhang, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018690 · Cited by 101
Spatio-Temporal Memory Attention for Image Captioning
Ji, J., Xu, C., Zhang, X., Wang, B., & Song, X. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3004729 · Cited by 83
LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
Tan, R., Xu, H., Saenko, K., & Plummer, B. A. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00213 · Cited by 107
Liu, G., Liao, Y., Wang, F., Zhang, B., Zhang, L., Liang, X., Wan, X., Li, S., Li, Z., Zhang, S., et al. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099165 · Cited by 80
A Medical Semantic-Assisted Transformer for Radiographic Report Generation
Wang, Z., Tang, M., Wang, L., Li, X., & Zhou, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16437-8_63 · Cited by 123
On hallucination and predictive uncertainty in conditional language generation
Xiao, Y., & Wang, W. Y. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.236 · Cited by 79
Expressing an image stream with a sequence of natural sentences
Park, C. C., & Kim, G. (2015) · Seoul National University Open Repository (Seoul National University) · Cited by 75
“What Happens If...” Learning to Predict the Effect of Forces in Images
Mottaghi, R., Rastegari, M., Gupta, A., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_17 · Cited by 77
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval
Mithun, N. C., Panda, R., Papalexakis, E. E., & Roy-Chowdhury, A. K. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240712 · Cited by 77
Phrase-based image caption generator with hierarchical LSTM network
Tan, Y. H., & Chan, C. S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.12.026 · Cited by 85
Multiview Concept Learning Via Deep Matrix Factorization
Zhao, W. Z., Xu, C. X., Guan, Z., & Liu, Y. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979532 · Cited by 79
Visual Grounding with Transformers
Du, Y., Fu, Z., Liu, Q., & Wang, Y. (2022) · 2022 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859880 · Cited by 71
Improving Cross-Modal Retrieval with Set of Diverse Embeddings
Kim, D.-W., Kim, N., & Kwak, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02243 · Cited by 115
Automatic Generation of Grounded Visual Questions
Zhang, S., Qu, L., You, S., Yang, Z., & Zhang, J. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/592 · Cited by 77
Image Caption Generation with Part of Speech Guidance
He, X., Shi, B., Bai, X., Xia, G.-S., Zhang, Z., & Dong, W. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.10.018 · Cited by 79
Knowledge Aware Semantic Concept Expansion for Image-Text Matching
Shi, B., Ji, L., Lu, P., Niu, Z., & Duan, N. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/720 · Cited by 81
Visual News: Benchmark and Challenges in News Image Captioning
Liu, F., Wang, Y., Wang, T., & Ordonez, V. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.542 · Cited by 75
M-FFN: multi-scale feature fusion network for image captioning
Prudviraj, J., Vishnu, C., & Mohan, C. K. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03463-x · Cited by 79
X2-VLM: All-in-One Pre-Trained Model for Vision-Language Tasks
Zeng, Y., Zhang, X., Li, H., Wang, J., Zhang, J., & Zhou, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3339661 · Cited by 166
Multimodal Alignment and Fusion: A Survey
Li, S., & Tang, H. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02667-1 · Cited by 295
The Technological Elements of Artificial Intelligence
Taddy, M. A. (2018) · National Bureau of Economic Research · doi:10.3386/w24301 · Cited by 120
Visual Question Answering with Question Representation Update (QRU)
Li, R., & Jia, J. (2016) · neural information processing systems · Cited by 97
Find and Focus: Retrieve and Localize Video Events with Natural Language Queries
Shao, D., Xiong, Y., Zhao, Y., Huang, Q., Qiao, Y., & Lin, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01240-3_13 · Cited by 83
Unsupervised Multi-Modal Neural Machine Translation
Su, Y., Fan, K., Bách, N., Kuo, C. ‐. J., & Huang, F. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01073 · Cited by 89
Multimodal Learning For Hateful Memes Detection
Zhou, Y., Chen, Z., & Yang, H. (2021) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew53276.2021.9455994 · Cited by 112
Combined scaling for zero-shot transfer learning
Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J., Chen, Y.-T., Luong, M.-T., Wu, Y., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126658 · Cited by 143
OmniMAE: Single Model Masked Pretraining on Images and Videos
Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01003 · Cited by 158
Analyzing human–human interactions: A survey
Stergiou, A., & Poppe, R. W. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.102799 · Cited by 96
Structured Matching for Phrase Localization
Wang, M., Azab, M., Kojima, N., Mihalcea, R. F., & Deng, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_42 · Cited by 86
Trip Outfits Advisor: Location-Oriented Clothing Recommendation
Zhang, X., Jia, J., Gao, K., Zhang, Y., Zhang, D., Li, J., & Tian, Q. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2696825 · Cited by 82
Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering
Jain, U., Schwing, A., & Lazebnik, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00603 · Cited by 76
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
Tan, H., & Bansal, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.162 · Cited by 75
UATVR: Uncertainty-Adaptive Text-Video Retrieval
Fang, B., Wu, W., Liu, C., Zhou, Y., Song, Y., Wang, W., Shu, X., Ji, X., & Wang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01262 · Cited by 143
The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance MatchingPreprint
Yu, Q., Chang, X., Song, Y.-Z., Xiang, T., & Hospedales, T. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08106 · Cited by 103
Online Multi-Target Tracking Using Recurrent Neural Networks
Milan, A., Rezatofighi, S. H., Dick, A., Reid, I., & Schindler, K. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11194 · Cited by 506
MemCap: Memorizing Style Knowledge for Image Captioning
Zhao, W., Wu, X., & Zhang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6998 · Cited by 75
Towards goal-oriented semantic signal processing: Applications and future challenges
Kalfa, M., Gok, M., Atalık, A., Tegin, B., Duman, T. M., & Arıkan, O. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2021.103134 · Cited by 94
Adaptive Path Selection for Dynamic Image Captioning
Xian, T., Li, Z., Tang, Z., & Ma, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3155795 · Cited by 79
DIFNet: Boosting Visual Information Flow for Image Captioning
Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., & Ji, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01749 · Cited by 99
THE AUGMENTED DESIGNER: A RESEARCH AGENDA FOR GENERATIVE AI-ENABLED DESIGN
Thoring, K., Huettemann, S., & Mueller, R. M. (2023) · Proceedings of the Design Society · doi:10.1017/pds.2023.335 · Cited by 104
Visual Madlibs: Fill in the blank Image Generation and Question Answering
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00278 · Cited by 70
Multi-Level Multimodal Common Semantic Space for Image-Phrase Grounding
Akbari, H., Karaman, S., Bhargava, S., Chen, B., Vondrick, C., & Chang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01276 · Cited by 95
HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation
Xiao, Q., Zheng, S., Wu, B., Xu, P., Qian, X., & Liang, Y. E. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00086 · Cited by 101
Geometry Attention Transformer with position-aware LSTMs for image captioning
Wang, C., Shen, Y., & Ji, L. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117174 · Cited by 82
Visuals to Text: A Comprehensive Review on Automatic Image Captioning
Ming, Y., Hu, N., Fan, C., Feng, F., Zhou, J., & Yu, H. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105734 · Cited by 73
Video2vec Embeddings Recognize Events When Examples Are Scarce
Habibian, A., Mensink, T., & Snoek, C. G. M. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2627563 · Cited by 73
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.