Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 901–1,000 of 1,041
Emotional Video Captioning With Vision-Based Emotion Interpretation Network
Song, P., Guo, D., Yang, X., Tang, S., & Wang, M. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3359045 · Cited by 90
IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction
Yang, J., Chen, Y., Du, S., Chen, B., & Príncipe, J. C. (2024) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2024.3359237 · Cited by 78
Inside Jokes: Identifying Humorous Cartoon Captions
Shahaf, D., Horvitz, E., & Mankoff, R. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783388 · Cited by 62
Barley yield and fertilization analysis from UAV imagery: a deep learning approach
Escalante, H. J., Rodríguez-Sánchez, S. V., Jiménez‐Lizárraga, M., Morales-Reyes, A., de la Calleja, J., & Vazquez, R. (2019) · International Journal of Remote Sensing · doi:10.1080/01431161.2019.1577571 · Cited by 62
SoDeep: A Sorting Deep Net to Learn Ranking Loss Surrogates
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01105 · Cited by 76
DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
Gu, X., Zhang, H., Zhang, D., & Kim, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/514 · Cited by 62
Write-a-video: computational video montage from themed text
Wang, M., Yang, G., Hu, S., Yau, S., & Shamir, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356520 · Cited by 92
Context-based information generation for managing UAV-acquired data using image captioning
Bang, S., & Kim, H. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103116 · Cited by 66
Recurrent Neural Networks for Semantic Instance Segmentation
Salvador, A., Bellver, M., Campos, V., Baradad, M., Marques, F., Torres, J., & Giro-i-Nieto, X. (2017) · arXiv · doi:10.48550/arxiv.1712.00617 · Cited by 62
A multimodal fusion approach for image captioning
Zhao, D., Chang, Z., & Guo, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.004 · Cited by 61
Show, Observe and Tell: Attribute-driven Attention Model for Image Captioning
Chen, H., Ding, G., Lin, Z., Zhao, S., & Han, J. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/84 · Cited by 61
LAM: Remote Sensing Image Captioning with Label-Attention Mechanism
Zhang, Z., Diao, W., Zhang, W., Yan, M., Gao, X., & Sun, X. (2019) · Remote Sensing · doi:10.3390/rs11202349 · Cited by 67
Transformer Reasoning Network for Image- Text Matching and Retrieval
Messina, N., Falchi, F., Esuli, A., & Amato, G. M. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413172 · Cited by 88
Neural attention for image captioning: review of outstanding methods
Zohourianshahzadi, Z., & Kalita, J. K. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10092-2 · Cited by 77
GLCM: Global–Local Captioning Model for Remote Sensing Image Captioning
Wang, Q., Huang, W. F., Zhang, X., & Li, X. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3222606 · Cited by 71
Cross-Modal Causal Representation Learning for Radiology Report Generation
Chen, W., Liu, Y., Wang, C., Zhu, J., Li, G., Liu, C., & Lin, L. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3568746 · Cited by 117
Few-Shot Object Recognition from Machine-Labeled Web Images
Xu, Z., Zhu, L., & Yang, Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.569 · Cited by 73
Semantic Text Summarization of Long Videos
Sah, S., Kulhare, S., Gray, A., Venugopalan, S., Prud’hommeaux, E. T., & Ptucha, R. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.115 · Cited by 61
Fine-Grained Image Retrieval: the Text/Sketch Input Dilemma
Song, J., Song, Y.-Z., Xiang, T., & Hospedales, T. M. (2017) · British Machine Vision Conference (BMVC) · doi:10.5244/c.31.45 · Cited by 75
Self-Critical N-Step Training for Image Captioning
Gao, J., Wang, S., Wang, S., Ma, S., & Gao, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00646 · Cited by 62
Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health
Ofli, F., Aytar, Y., Weber, I. G., al Hammouri, R., & Torralba, A. B. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052663 · Cited by 69
Cops-Ref: A New Dataset and Task on Compositional Referring Expression Comprehension
Chen, Z., Wang, P., Ma, L., Wong, K.-Y. K., & Wu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01010 · Cited by 59
Vision-Enhanced and Consensus-Aware Transformer for Image Captioning
Cao, S., An, G., Zheng, Z., & Wang, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3178844 · Cited by 65
MCRN: A Multi-source Cross-modal Retrieval Network for remote sensing
Yuan, Z., Zhang, W., Tian, C., Mao, Y., Zhou, R., Wang, H., Fu, K., & Sun, X. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.103071 · Cited by 68
A Brand New Dance Partner: Music-Conditioned Pluralistic Dancing Controlled by Multiple Dance Genres
Kim, J., Oh, H., Kim, S., Tong, H., & Lee, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00348 · Cited by 89
Intention Oriented Image Captions With Guiding Objects
Zheng, Y., Li, Y., & Wang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00859 · Cited by 60
Multiscale Object Detection from Drone Imagery Using Ensemble Transfer Learning
Walambe, R. A., Marathe, A., & Kotecha, K. V. (2021) · Drones · doi:10.3390/drones5030066 · Cited by 73
Joint Global and Co-Attentive Representation Learning for Image-Sentence Retrieval
Wang, S., Chen, Y., Zhuo, J., Huang, Q., & Tian, Q. C. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240535 · Cited by 63
Integration of textual cues for fine-grained image captioning using deep CNN and LSTM
Gupta, N., & Jalal, A. S. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04515-z · Cited by 65
A survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59
Retrieval-Augmented Transformer for Image Captioning
Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2022) · International Conference on Content-based Multimedia Indexing · doi:10.1145/3549555.3549585 · Cited by 49
MeaCap: Memory-Augmented Zero-shot Image Captioning
Zeng, Z.-Q., Xie, Y., Zhang, H., Chen, C., Chen, B., & Wang, Z. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01337 · Cited by 93
Liu, Y., Fu, J., Mei, T., & Chen, C. W. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10760 · Cited by 63
CNN Fixations: An Unraveling Approach to Visualize the Discriminative Image Regions
Mopuri, K. R., Garg, U., & Venkatesh Babu, R. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2881920 · Cited by 59
Improving variational autoencoder with deep feature consistent and generative adversarial training
Hou, X., Sun, K., Shen, L., & Qiu, G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.03.013 · Cited by 69
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
Ma, S., Cui, L., Dai, D., Wei, F., & Sun, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016810 · Cited by 81
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning
Aneja, J., Agrawal, H., Batra, D., & Schwing, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00436 · Cited by 57
Alignment for Advanced Machine Learning Systems
Taylor, J., Yudkowsky, E., LaVictoire, P., & Critch, A. (2020) · Ethics of Artificial Intelligence · doi:10.1093/oso/9780190905033.003.0013 · Cited by 59
Referring Expression Object Segmentation with Caption-Aware ConsistencyPreprint
Chen, Y., Tsai, Y., Wang, T., Lin, Y., & Yang, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04748 · Cited by 101
Average biased ReLU based CNN descriptor for improved face retrieval
Dubey, S. R., & Chakraborty, S. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10269-x · Cited by 63
Robust Unsupervised Cross-modal Hashing for Multimedia Retrieval
Cheng, M., Jing, L., & Ng, M. K. (2020) · ACM Transactions on Information Systems · doi:10.1145/3389547 · Cited by 60
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Zhou, Y., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00414 · Cited by 119
Fang, W., Love, P. E. D., Ding, L., Xu, S., Kong, T., & Li, H. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3093166 · Cited by 62
A parallel-fusion RNN-LSTM architecture for image caption generation
Wang, M., Song, L., Yang, X., & Luo, C. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7533201 · Cited by 58
Towards Personalized Image Captioning via Multimodal Memory Networks
Park, C. C., Kim, B., & Kim, G. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2824816 · Cited by 65
Image Captioning with Scene-graph Based Semantic Concepts
Gao, L., Wang, B., & Wang, W. (2018) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3195106.3195114 · Cited by 61
Multi-modal semantic autoencoder for cross-modal retrieval
Wu, Y., Wang, S., & Huang, Q. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.11.042 · Cited by 63
Sequence-to-sequence Bangla Sentence Generation with LSTM Recurrent Neural Networks
Islam, M. S., Mousumi, S. S. S., Abujar, S., & Hossain, S. A. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.05.026 · Cited by 58
Deep Learning in Natural Language Processing: A State-of-the-Art Survey
Chai, J., & Li, A. (2019) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc48188.2019.8949185 · Cited by 60
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
Ni, M., Huang, H., Su, L., Cui, E., Bharti, T., Wang, L., Zhang, D., & Duan, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00397 · Cited by 57
Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory
Vasudevan, A. B., Dai, D., & Gool, L. V. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01374-3 · Cited by 101
Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval
Ge, X., Chen, F., Jose, J. M., Ji, Z., Wu, Z., & Liu, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475634 · Cited by 60
MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text Retrieval
Feng, D., He, X., & Peng, Y. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3580501 · Cited by 78
Deep Learning for Information Retrieval
Li, H., & Lu, Z. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2914800 · Cited by 57
Teaching Compositionality to CNNs
Stone, A., Wang, H., Stark, M., Liu, Y., Phoenix, D. S., & George, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.85 · Cited by 55
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech
Harwath, D. F., Chuang, G., & Glass, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462396 · Cited by 69
Protecting Intellectual Property of Language Generation APIs with Lexical Watermark
He, X., Xu, Q., Lyu, L., Wu, F., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21321 · Cited by 56
Derin Öğrenme Modelleri ve Uygulama Alanlarına İlişkin Bir Derleme
Doğan, F., & Türkoğlu, İ. (2019) · DÜMF Mühendislik Dergisi · doi:10.24012/dumf.411130 · Cited by 103
Show, Tell, and Polish: Ruminant Decoding for Image Captioning
Guo, L., Liu, J., Lu, S., & Lu, H. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2951226 · Cited by 55
GraDual: Graph-based Dual-modal Representation for Image-Text Matching
Long, S., Han, S. C., Wan, X., & Poon, J. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00252 · Cited by 76
Adaptive Semantic-Enhanced Transformer for Image Captioning
Zhang, J., Fang, Z., Sun, H., & Wang, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3185320 · Cited by 65
Hypertuned Convolutional Neural Network Residual Model Based Content Based Image Retrival System
Singh, A. D., Dixit, A., & Singh, B. K. (2022) · International Conference on Fourth Industrial Revolution Based Technology and Practices (ICFIRTP) · doi:10.1109/icfirtp56122.2022.10059435 · Cited by 61
A Prior Instruction Representation Framework for Remote Sensing Image-text Retrieval
Pan, J., Ma, Q., & Bai, C. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612374 · Cited by 89
Memory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation
Divya, P., Yenduri, S., Vishnu, C., Mohan, C. K., & Chen, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3371894 · Cited by 65
Neural Image Caption Generation with Weighted Training and Reference
Ding, G., Chen, M., Zhao, S., Chen, H., Han, J., & Liu, Q. (2018) · Cognitive Computation · doi:10.1007/s12559-018-9581-x · Cited by 57
COMIC: Toward A Compact Image Captioning Model With Attention
Tan, J. H., Chan, C. S., & Chuah, J. H. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2904878 · Cited by 60
Storytelling from an Image Stream Using Scene Graphs
Wang, R., Wei, Z., Li, P., Zhang, Q., & Huang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6455 · Cited by 82
SMAN: Stacked Multimodal Attention Network for Cross-Modal Image–Text Retrieval
Ji, Z., Wang, H., Han, J., & Pang, Y. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2985716 · Cited by 59
Zhu, K., Zhang, N., Ying, S., & Zhu, D. (2020) · IET Software · doi:10.1049/iet-sen.2019.0278 · Cited by 60
Integrating Scene Semantic Knowledge into Image Captioning
Wei, H., Li, Z., Huang, F., Zhang, C., Ma, H., & Shi, Z. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3439734 · Cited by 55
Ho, S., Zhang, W., Young, W., Buchholz, M., Al-Jufout, S., Dajani, K. F., Bian, L., & Mozumdar, M. (2021) · IEEE Access · doi:10.1109/access.2021.3105362 · Cited by 62
Pan, J., Ma, Q., & Bai, C. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592236 · Cited by 57
Global-local fusion based on adversarial sample generation for image-text matching
Huang, S., Fu, W., Zhang, Z., & Liu, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102084 · Cited by 56
A Joint Introduction to Natural Language Processing and to Deep Learning
Deng, L., & Liu, Y. (2018) · Springer eBooks · doi:10.1007/978-981-10-5209-5_1 · Cited by 73
Multi-Task Learning of Hierarchical Vision-Language Representation
Nguyen, D.-K., & Okatani, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01074 · Cited by 55
Exact Adversarial Attack to Image Captioning via Structured Output Learning With Latent Variables
Xu, Y., Wu, B., Shen, F., Fan, Y., Zhang, Y., Shen, H. T., & Liu, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00426 · Cited by 56
Image Captioning with Object Detection and Localization
Yang, Z., Zhang, Y., Rehman, S. U., & Huang, Y. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71589-6_10 · Cited by 62
Tripathi, S., Singh, S. K., & Lee, H. K. (2020) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2020.101838 · Cited by 59
Shin, A., Ishii, M., & Narihira, T. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-021-01547-8 · Cited by 65
Scaling Vision-Language Models with Sparse Mixture of Experts
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.758 · Cited by 148
Generating Descriptions with Grounded and Co-referenced People
Rohrbach, A., Rohrbach, M., Tang, S., Oh, S. J., & Schiele, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.447 · Cited by 63
Wearable Motion Sensor Based Analysis of Swing Sports
Anand, A., Sharma, M., Srivastava, R., Kaligounder, L., & Prakash, D. (2017) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2017.0-149 · Cited by 61
A Deep Neural Network for Oil Spill Semantic Segmentation in Sar Images
Orfanidis, G., Ioannidis, K., Avgerinakis, K., Vrochidis, S., & Kompatsiaris, I. Y. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451113 · Cited by 54
Boosted Attention: Leveraging Human Attention for Image Captioning
Chen, S., & Zhao, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_5 · Cited by 59
Vision-to-Language Tasks Based on Attributes and Attention Mechanism
Li, X., Yuan, A., & Lu, X. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2914351 · Cited by 62
Kim, H., Lee, K. M., Kim, E. J., & Lee, J. S. (2019) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims.2019.05.15 · Cited by 58
ADVISE: Symbolism and External Knowledge for Decoding Advertisements
Ye, K., & Kovashka, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_51 · Cited by 67
Image captioning via hierarchical attention mechanism and policy gradient optimization
Yan, S., Xie, Y., Wu, F., Smith, J. S., Lu, W., & Zhang, B. (2019) · Signal Processing · doi:10.1016/j.sigpro.2019.107329 · Cited by 67
Multimodal Entity Linking for Tweets
Adjali, O., Besançon, R., Ferret, O., Borgne, H. L., & Grau, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_31 · Cited by 95
Multimodal Attention for Neural Machine TranslationPreprint
Çağlayan, O., Barrault, L., & Bougares, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03976 · Cited by 87
A systematic deep learning model selection for p300-based brain-computer interfaces
Abibullaev, B., & Zollanvari, A. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3051136 · Cited by 53
Manufacturing an Artificial Intelligence Revolution
Katz, Y. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3078224 · Cited by 53
Training LSTM Networks With Resistive Cross-Point Devices
Gokmen, T., Rasch, M. J., & Haensch, W. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00745 · Cited by 53
PFAN++: Bi-Directional Image-Text Retrieval With Position Focused Attention Network
Wang, Y., Yang, H., Bai, X., Qian, X., Ma, L., Lu, J. J., Li, B., & Fan, X. N. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3024822 · Cited by 63
SAMoSA: Sensing Activities with Motion and Subsampled Audio
Mollyn, V., Ahuja, K., Verma, D., Harrison, C., & Goel, M. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3550284 · Cited by 82
ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing
Zeng, Z., Zhang, H., Lu, R., Wang, D., Chen, B., & Wang, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02247 · Cited by 83
Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation
Che, C., Lin, Q., Zhao, X., Huang, J., & Yu, L. (2023) · International Conference on Big Data Technologies (ICBDT) · doi:10.1145/3627377.3627442 · Cited by 90
Bi-Prediction: Pedestrian Trajectory Prediction Based on Bidirectional LSTM Classification
Xue, H., Huynh, D. Q., & Reynolds, M. A. (2017) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2017.8227412 · Cited by 54
Video Storytelling: Textual Summaries for Events
Li, J., Wong, Y., Zhao, Q., & Kankanhalli, M. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2930041 · Cited by 68
Qu, R., Xu, G., Ding, C., Jia, W., & Sun, M. (2019) · IEEE Access · doi:10.1109/access.2019.2950387 · Cited by 61
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.