Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 401–500 of 1,041
Binary convolutional neural network on RRAM
Tang, T., Xia, L., Li, B., Wang, Y., & Yang, H. (2017) · 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/aspdac.2017.7858419 · Cited by 176
Partially View-aligned Representation Learning with Noise-robust Contrastive Loss
Yang, M., Li, Y., Huang, Z., Liu, Z., Hu, P., & Peng, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00119 · Cited by 171
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.488 · Cited by 170
Cross-Modal and Hierarchical Modeling of Video and Text
Zhang, B., Hu, H., & Sha, F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_23 · Cited by 173
Inverse Cooking: Recipe Generation From Food Images
Salvador, A., Drozdzal, M., Giro-i-Nieto, X., & Romero, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01070 · Cited by 168
Fu, K., Jin, J., Cui, R., Sha, F., & Zhang, C. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2642953 · Cited by 176
Recurrent Modeling of Interaction Context for Collective Activity Recognition
Wang, M., Ni, B., & Yang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.783 · Cited by 170
Causal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175
Deep learning: Evolution and expansion
Wason, R. (2018) · Cognitive Systems Research · doi:10.1016/j.cogsys.2018.08.023 · Cited by 174
Dense Captioning with Joint Inference and Visual Context
Yang, L., Tang, K., Yang, J., & Li, L.-J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.214 · Cited by 167
Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings
Carvalho, M., Cadène, R., Picard, D., Soulier, L., Thome, N., & Cord, M. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210036 · Cited by 171
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
Harwath, D., Recasens, A., Surís, D., Chuang, G., Torralba, A., & Glass, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_40 · Cited by 165
Linking Image and Text with 2-Way Nets
Eisenschtat, A., & Wolf, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.201 · Cited by 171
Fathom: reference workloads for modern deep learning methods
Adolf, R., Rama, S., Reagen, B., Wei, G.-Y., & Brooks, D. (2016) · 2016 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc.2016.7581275 · Cited by 169
RNN Fisher Vectors for Action Recognition and Image Annotation
Lev, G., Sadeh, G., Klein, B., & Wolf, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_50 · Cited by 164
Weakly-Supervised Video Moment Retrieval via Semantic Completion Network
Lin, Z., Zhao, Z., Zhang, Z., Wang, Q., & Liu, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6820 · Cited by 166
End-to-End Transformer Based Model for Image Captioning
Wang, Y., Xu, J., & Sun, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20160 · Cited by 230
Zero-Shot Visual Recognition via Bidirectional Latent Embedding
Wang, Q., & Chen, K. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1027-5 · Cited by 167
Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues
Plummer, B. A., Mallya, A., Cervantes, C. M., Hockenmaier, J., & Lazebnik, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.213 · Cited by 161
Learning-Based Energy-Efficient Data Collection by Unmanned Vehicles in Smart Cities
Zhang, B., Liu, C. H., Tang, J., Xu, Z., Ma, J., & Wang, W. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2783439 · Cited by 166
Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding
Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.208 · Cited by 161
Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
Park, C. C., Kim, B., & Kim, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.681 · Cited by 163
Engaging Image Captioning via Personality
Shuster, K., Humeau, S., Hu, H., Bordes, A., & Weston, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01280 · Cited by 158
Ultrasound Standard Plane Detection Using a Composite Neural Network Framework
Chen, H., Wu, L., Dou, Q., Qin, J., Li, S., Cheng, J.-Z., Ni, D., & Heng, P.-A. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2685080 · Cited by 164
Short-term traffic flow prediction based on optimized deep learning neural network: PSO-Bi-LSTM
Bharti, Redhu, P., & Kumar, K. (2023) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2023.129001 · Cited by 161
DSD: Dense-Sparse-Dense Training for Deep Neural Networks
Han, S., Pool, J., Narang, S., Mao, H., Gong, E., Tang, S., Elsen, E., Vajda, P., Paluri, M., Tran, J. J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.04381 · Cited by 146
Wu, H., Mao, J., Zhang, Y., Jiang, Y., Li, L., Sun, W., & Ma, W.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00677 · Cited by 155
Visual Sentiment Analysis by Attending on Local Image Regions
You, Q., Jin, H., & Luo, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10501 · Cited by 158
RGB-D object detection and semantic segmentation for autonomous manipulation in clutter
Schwarz, M., Milan, A., Periyasamy, A. S., & Behnke, S. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917713117 · Cited by 159
Learning a Recurrent Residual Fusion Network for Multimodal Matching
Liu, Y., Guo, Y., Bakker, E. M., & Lew, M. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.442 · Cited by 159
ArtEmis: Affective Language for Visual Art
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01140 · Cited by 161
Image Caption with Global-Local Attention
Li, L., Tang, S., Deng, L., Zhang, Y., & Tian, Q. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11236 · Cited by 155
Automatic Understanding of Image and Video Advertisements
Hussain, Z., Zhang, M., Zhang, X., Ye, K., Thomas, C., Agha, Z., Ong, N., & Kovashka, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.123 · Cited by 232
Predicting the Risk of Heart Failure With EHR Sequential Data Modeling
Jin, B., Che, C., Liu, Z., Zhang, S., Yin, X., & Wei, X. (2018) · IEEE Access · doi:10.1109/access.2017.2789324 · Cited by 160
Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner
Chen, T.-H., Liao, Y.-H., Chuang, C.-Y., Hsu, W.-T., Fu, J., & Sun, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.64 · Cited by 158
Capatina, A., Kachour, M., Lichy, J., Micu, A., Micu, A.-E., & Codignola, F. (2020) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2019.119794 · Cited by 159
SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences
Wu, S.-C., Wald, J., Tateno, K., Navab, N., & Tombari, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00743 · Cited by 158
An Empirical Study of Language CNN for Image Captioning
Gu, J., Wang, G., Cai, J., & Chen, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.138 · Cited by 150
Image-Text Embedding Learning via Visual and Textual Semantic Reasoning
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148470 · Cited by 192
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Look Back and Predict Forward in Image Captioning
Qin, Y., Du, J., Zhang, Y., & Lu, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00856 · Cited by 147
Monocular Object Instance Segmentation and Depth Ordering with CNNs
Zhang, Z., Schwing, A. G., Fidler, S., & Urtasun, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.300 · Cited by 152
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J., & Loy, C. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_20 · Cited by 169
Bimodal Modelling of Source Code and Natural Language
Allamanis, M., Tarlow, D., Gordon, A. D., & Wei, Y. (2015) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 215
VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis
Truong, Q.-T., & Lauw, H. W. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301305 · Cited by 187
Zamir, A., Wu, T.-L., Sun, L., Shen, W. B., Shi, B. E., Malik, J., & Savarese, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.196 · Cited by 178
A comprehensive overview of knowledge graph completion
Shen, T., Zhang, F., & Cheng, J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109597 · Cited by 209
Wen, T.-H., Gasic, M., Kim, D., Mrksic, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4639 · Cited by 151
Fashion Style in 128 Floats: Joint Ranking and Classification Using Weak Data for Feature Extraction
Simo‐Serra, E., & Ishikawa, H. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.39 · Cited by 165
RETRACTED ARTICLE: LSTM Model for Prediction of Heart Failure in Big Data
Maragatham, G., & Devi, S. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1243-3 · Cited by 175
RetractedSee-Through-Text Grouping for Referring Image Segmentation
Chen, D.-J., Jia, S., Lo, Y.-C., Chen, H.-T., & Liu, T.-L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00755 · Cited by 181
GRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features
Nguyen, V.-Q., Suganuma, M., & Okatani, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_10 · Cited by 210
Towards Context-Aware Interaction Recognition for Visual Relationship Detection
Zhuang, B., Liu, L., Shen, C., & Reid, I. D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.71 · Cited by 177
Automatic Human Sleep Stage Scoring Using Deep Neural Networks
Malafeev, A., Laptev, D., Bauer, S., Omlin, X., Wierzbicka, A., Wichniak, A., Jernajczyk, W., Riener, R., Buhmann, J., & Achermann, P. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00781 · Cited by 151
Chang, C., Huang, D.-A., Sui, Y.-N., Fei-Fei, L., & Niebles, J. C. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00366 · Cited by 186
Self-citationAutomated Crowdturfing Attacks and Defenses in Online Review Systems
Yao, Y., Viswanath, B., Cryan, J., Zheng, H., & Zhao, B. Y. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133990 · Cited by 150
Deep multimodal semantic embeddings for speech and images
Harwath, D. F., & Glass, J. R. (2015) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404800 · Cited by 192
Bačanin, N., Bezdan, T., Tuba, E., Strumberger, I., & Tuba, M. (2020) · Algorithms · doi:10.3390/a13030067 · Cited by 170
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders
Messina, N., Amato, G., Esuli, A., Falchi, F., Gennaro, C., & Marchand-Maillet, S. (2021) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3451390 · Cited by 149
Palani, B., Elango, S., & Vignesh Viswanathan, K. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11782-3 · Cited by 163
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Mao, J., Wei, X., Yang, Y., Wang, J., Huang, Z., & Yuille, A. L. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.291 · Cited by 148
Visual Madlibs: Fill in the Blank Description Generation and Question Answering
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.283 · Cited by 148
Comprehension-Guided Referring Expressions
Luo, R., & Shakhnarovich, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.333 · Cited by 144
Context-Aware Visual Policy Network for Fine-Grained Image Captioning
Zha, Z.-J., Liu, D., Zhang, H., Zhang, Y., & Wu, F. H. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2909864 · Cited by 168
Exploring Nearest Neighbor Approaches for Image Captioning
Devlin, J., Gupta, S., Girshick, R., Mitchell, M., & Zitnick, C. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04467 · Cited by 143
Weakly Supervised Dense Video Captioning
Shen, Z., Li, J., Su, Z., Li, M., Chen, Y., Jiang, Y.-G., & Xue, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.548 · Cited by 141
A Recurrent Encoder-Decoder Network for Sequential Face Alignment
Peng, X., Feris, R. S., Wang, X., & Metaxas, D. N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_3 · Cited by 146
More Grounded Image Captioning by Distilling Image-Text Matching Model
Zhou, Y., Wang, M., Liu, D., Hu, Z., & Zhang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00483 · Cited by 147
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 144
Generalized Deep Transfer Networks for Knowledge Propagation in Heterogeneous Domains
Tang, J., Shu, X., Li, Z., Qi, G.-J., & Wang, J. (2016) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/2998574 · Cited by 144
Top-Down Visual Saliency Guided by Captions
Ramanishka, V., Das, A., Zhang, J., & Saenko, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.334 · Cited by 140
Language-Based Image Editing with Recurrent Attentive Models
Chen, J., Shen, Y., Gao, J., Liu, J., & Liu, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00909 · Cited by 171
Parallel Attention: A Unified Framework for Visual Object Discovery Through Dialogs and Queries
Zhuang, B., Wu, Q., Shen, C., Reid, I., & Hengel, A. V. D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00447 · Cited by 134
CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback
Lee, S., Kim, D.-W., & Han, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00086 · Cited by 215
Monkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., & Bai, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02527 · Cited by 136
Learning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00608 · Cited by 135
Fast, Diverse and Accurate Image Captioning Guided by Part-Of-Speech
Deshpande, A., Aneja, J., Wang, L., Schwing, A. G., & Forsyth, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01095 · Cited by 139
Exploring Visual Relationship for Image CaptioningPreprint
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07041 · Cited by 164
Hierarchical Attention Network for Image Captioning
Wang, W., Chen, Z., & Hu, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018957 · Cited by 124
Image Captioning Through Image Transformer
He, S., Liao, W., Tavakoli, H. R., Yang, M. Y., Rosenhahn, B., & Pugeault, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_10 · Cited by 149
I am Robot: (Deep) Learning to Break Semantic Image CAPTCHAs
Sivakorn, S., Polakis, I., & Keromytis, A. D. (2016) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp.2016.37 · Cited by 145
Situation Recognition with Graph Neural Networks
Li, R., Tapaswi, M., Liao, R., Jia, J., Urtasun, R., & Fidler, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.448 · Cited by 133
Comprehensive Image Captioning via Scene Graph Decomposition
Zhong, Y., Wang, L., Chen, J., Yu, D., & Li, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_13 · Cited by 140
AMFB: Attention based multimodal Factorized Bilinear Pooling for multimodal Fake News Detection
Kumari, R., & Ekbal, A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115412 · Cited by 164
Towards local visual modeling for image captioning
Ma, Y., Ji, J., Sun, X., Zhou, Y., & Ji, R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109420 · Cited by 161
Extending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 156
Comprehending and Ordering Semantics for Image Captioning
Li, Y., Pan, Y., Yao, T., & Mei, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01746 · Cited by 174
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R. J., Darrell, T. J., & Saenko, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00487 · Cited by 136
Referring Expression Generation and Comprehension via Attributes
Liu, J., Wang, L., & Yang, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.520 · Cited by 163
Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
Liu, X., Li, H., Shao, J., Chen, D., & Wang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_21 · Cited by 143
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
Chen, D. Z., Gholami, A., Niesner, M., & Chang, A. X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00321 · Cited by 137
Cross-Modal Retrieval via Deep and Bidirectional Representation Learning
He, Y., Xiang, S., Kang, C., Wang, J., & Pan, C. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2558463 · Cited by 134
Ma, L., Li, Y., Li, J., Tan, W., Yu, Y., & Chapman, M. A. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2961060 · Cited by 148
An Efficient PointLSTM for Point Clouds Based Gesture Recognition
Min, Y., Zhang, Y., Chai, X., & Chen, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00580 · Cited by 138
Lu, W., Tao, C., Li, H., Qi, J., & Li, Y. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2021.112830 · Cited by 149
Smallcap: Lightweight Image Captioning Prompted with Retrieval Augmentation
Ramos, R., Martins, B., Elliott, D., & Kementchedjhieva, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00278 · Cited by 128
Latent Intention Dialogue ModelsPreprint
Wen, T.-H., Miao, Y., Blunsom, P., & Young, S. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10229 · Cited by 142
Morris, M. R., Zolyomi, A., Yao, C., Bahram, S., Bigham, J. P., & Kane, S. K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858116 · Cited by 131
Multitask Learning for Cross-Domain Image Captioning
Yang, M., Zhao, W., Xu, W. H., Feng, Y., Zhao, Z., Chen, X., & Lei, K. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2869276 · Cited by 141
Connecting Touch and Vision via Cross-Modal Prediction
Li, Y., Zhu, J.-Y., Tedrake, R., & Torralba, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01086 · Cited by 131
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.