Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 801–900 of 1,041
Image Captioning Based on Deep Neural Networks
Liu, S., Bai, L., Hu, Y., & Wang, H. (2018) · MATEC Web of Conferences · doi:10.1051/matecconf/201823201052 · Cited by 83
High-Quality Image Captioning With Fine-Grained and Semantic-Guided Visual Attention
Zhang, Z., Wu, Q., Wang, Y., & Chen, F. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2888822 · Cited by 74
Cross-Modal Image-Text Retrieval with Semantic Consistency
Chen, H., Ding, G., Lin, Z., Zhao, S., & Han, J. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351055 · Cited by 78
Detection and classification of supernova gravitational wave signals: A deep learning approach
Chan, M., Heng, I. S., & Messenger, C. (2020) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.102.043022 · Cited by 81
Show, Edit and Tell: A Framework for Editing Image Captions
Sammani, F., & Melas-Kyriazi, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00486 · Cited by 84
Radial Graph Convolutional Network for Visual Question Generation
Xu, X., Wang, T., Yang, Y., Hanjalić, A., & Shen, H. T. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2986029 · Cited by 84
Federated Learning on Multimodal Data: A Comprehensive Survey
Lin, Y., Gao, Y., Gong, M., Zhang, S., Zhang, Y., & Li, Z. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1398-0 · Cited by 96
Bidirectional Long-Short Term Memory for Video Description
Bin, Y., Yang, Y., Shen, F., Xu, X., & Shen, H. T. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967258 · Cited by 87
Definition Modeling: Learning to Define Word Embeddings in Natural Language
Noraset, T., Liang, C., Birnbaum, L., & Downey, D. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.10996 · Cited by 150
Coreset-Based Neural Network Compression
Dubey, A., Chatterjee, M., & Ahuja, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_28 · Cited by 73
Zhou, L., Xu, C., Koch, P. A., & Corso, J. J. (2017) · Thematic Workshops of ACM Multimedia · doi:10.1145/3126686.3126717 · Cited by 73
Monarch Butterfly Optimization Based Convolutional Neural Network Design
Bačanin, N., Bezdan, T., Tuba, E., Strumberger, I., & Tuba, M. (2020) · Mathematics · doi:10.3390/math8060936 · Cited by 77
Dissecting the Meme Magic: Understanding Indicators of Virality in Image Memes
Ling, C., AbuHilal, I., Blackburn, J., Cristofaro, E. D., Zannettou, S., & Stringhini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449155 · Cited by 87
Human-like Controllable Image Captioning with Verb-specific Semantic Roles
Chen, L., Jiang, Z., Xiao, J., & Liu, W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01657 · Cited by 74
Bloom’s Learning Outcomes’ Automatic Classification Using LSTM and Pretrained Word Embeddings
Shaikh, S., Daudpotta, S. M., & Imran, A. S. (2021) · IEEE Access · doi:10.1109/access.2021.3106443 · Cited by 70
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
Fei, J., Wang, T., Zhang, J., He, Z., Wang, C., & Zheng, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00291 · Cited by 128
Foundations of spatial perception for robotics: Hierarchical representations and real-time systems
Hughes, N., Chang, Y., Hu, S., Talak, R., Abdulhai, R., Strader, J., & Carlone, L. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241229725 · Cited by 143
Zhang, Z., Chen, P., Sapkota, M., & Yang, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_37 · Cited by 82
Deep Learning in Automatic Sleep Staging With a Single Channel Electroencephalography
Fu, M., Wang, Y., Chen, Z., Li, J., Xu, F., Liu, X., & Hou, F. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.628502 · Cited by 83
Liapis, A., Yannakakis, G. N., Nelson, M. J., Preuss, M., & Bidarra, R. (2019) · IEEE Transactions on Games · doi:10.1109/tg.2018.2870876 · Cited by 70
Interpretable Multimodal Retrieval for Fashion Products
Liao, L., He, X., Zhao, B., Ngo, C., & Chua, T. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240646 · Cited by 82
Recurrent Relational NetworksPreprint
Palm, R. B., Paquet, U., & Winther, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08028 · Cited by 98
Real-World Image Datasets for Federated LearningPreprint
Luo, J., Wu, X., Luo, Y., Huang, A., Huang, Y., Liu, Y., & Yang, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11089 · Cited by 131
XGPT: Cross-modal Generative Pre-Training for Image Captioning
Xia, Q., Huang, H., Duan, N., Zhang, D., Ji, L., Sui, Z., Cui, E., Bharti, T., & Zhou, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-88480-2_63 · Cited by 70
Visual-textual sentiment classification with bi-directional multi-level attention networks
Xu, J. J., Huang, F., Zhang, X., Wang, S., Li, C., Li, Z., & He, Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.04.018 · Cited by 75
Shapeglot: Learning Language for Shape Differentiation
Achlioptas, P., Guibas, L., Goodman, N., Fan, J., & Hawkins, R. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00903 · Cited by 70
LSTM, VADER and TF-IDF based Hybrid Sentiment Analysis Model
Chiny, M., Chihab, M., Bencharef, O., & Chihab, Y. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120730 · Cited by 101
A Simple Baseline for Audio-Visual Scene-Aware Dialog
Schwartz, I., Schwing, A. G., & Hazan, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01283 · Cited by 95
Progressively Parsing Interactional Objects for Fine Grained Action Detection
Ni, B., Yang, X., & Gao, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.116 · Cited by 69
CNN+CNN: Convolutional Decoders for Image CaptioningPreprint
Wang, Q., & Chan, A. B. (2018) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1805.09019 · Cited by 95
TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal
Zhu, H., Vial, R., & Lu, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.619 · Cited by 68
A Systematic Literature Review on Image Captioning
Staniūtė, R., & Šešok, D. (2019) · Applied Sciences · doi:10.3390/app9102024 · Cited by 83
Low Bit-Width Convolutional Neural Network on RRAM
Cai, Y., Tang, T., Xia, L., Li, B., Wang, Y., & Yang, H. (2019) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2019.2917852 · Cited by 71
Visually Grounded Neural Syntax Acquisition
Shi, H., Mao, J., Gimpel, K., & Livescu, K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1180 · Cited by 90
ACMM: Aligned Cross-Modal Memory for Few-Shot Image and Sentence Matching
Huang, Y., & Wang, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00587 · Cited by 81
Expressing Objects Just Like Words: Recurrent Visual Embedding for Image-Text Matching
Chen, T., & Luo, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6631 · Cited by 70
Length-Controllable Image Captioning
Deng, C., Ding, N., Tan, M., & Wu, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_42 · Cited by 70
Boosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph
Zhao, W., & Wu, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301279 · Cited by 96
A Novel SVM-Based Decoder for Remote Sensing Image Captioning
Hoxha, G., & Melgani, F. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3105004 · Cited by 89
Chen, X., Song, L., & Yang, X. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2239260 · Cited by 74
A Hierarchical Grocery Store Image Dataset With Visual and Semantic Labels
Klasson, M., Zhang, C., & Kjellström, H. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00058 · Cited by 92
Integrating Long Short-Term Memory and Genetic Algorithm for Short-Term Load Forecasting
Santra, A. S., & Lin, J.-L. (2019) · Energies · doi:10.3390/en12112040 · Cited by 69
Multilingual Image Description with Neural Sequence ModelsPreprint
Elliott, D., Frank, S., & Hasler, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.04709 · Cited by 100
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
Zhou, L., Louis, N., & Corso, J. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.02834 · Cited by 62
Yang, M., Liu, J., Shen, Y., Zhao, Z., Chen, X., Wu, Q., & Li, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3028651 · Cited by 71
Dual self-attention with co-attention networks for visual question answering
Liu, Y., Zhang, X., Zhang, Q., Li, C., Huang, F., Tang, X., & Li, Z. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107956 · Cited by 76
Text to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 80
Real-time structural damage assessment using LSTM networks: regression and classification approaches
Sharma, S., & Sen, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07773-6 · Cited by 78
Textual Context-Aware Dense Captioning With Diverse Words
Shao, Z., Han, J., Debattista, K., & Pang, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3241517 · Cited by 69
Bhooshan, R. S., Kumaraswamy, S., & Sasidharan, S. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00693-9 · Cited by 77
Expressive Text-to-Image Generation with Rich Text
Ge, S., Park, T., Zhu, J.-Y., & Huang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00694 · Cited by 136
Reale-Nosei, G., Amador-Domínguez, E., & Serrano, E. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103264 · Cited by 88
Fine-Grained Video Captioning for Sports Narrative
Yu, H., Cheng, S., Ni, B., Wang, M., Zhang, J., & Yang, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00629 · Cited by 71
Diverse and Coherent Paragraph Generation from Images
Chatterjee, M., & Schwing, A. G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_45 · Cited by 70
Recurrent neural network-based semantic variational autoencoder for Sequence-to-sequence learning
Jang, M., Seo, S., & Kang, P. (2019) · Information Sciences · doi:10.1016/j.ins.2019.03.066 · Cited by 75
Guiding Long-Short Term Memory for Image Caption GenerationPreprint
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.04942 · Cited by 67
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
Harwath, D. F., Recasens, A., Surís, D., Chuang, G., Torralba, A. B., & Glass, J. R. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01205-0 · Cited by 92
Making History Matter: History-Advantage Sequence Training for Visual Dialog
Yang, T., Zha, Z.-J., & Zhang, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00265 · Cited by 88
A New Image Captioning Approach for Visually Impaired People
Makav, B., & Kılıç, V. (2019) · National Conference on Electrical and Electronics Engineering (ELECO) · doi:10.23919/eleco47770.2019.8990630 · Cited by 70
Auto-encoding and Distilling Scene Graphs for Image Captioning
Yang, X., Zhang, H., & Cai, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3042192 · Cited by 77
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
Tian, Y., Hu, D., & Xu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00277 · Cited by 108
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Hu, X., Yin, X., Lin, K., Zhang, L., Gao, J., Wang, L., & Liu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16249 · Cited by 64
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 80
Deep Learning in Diverse Intelligent Sensor Based Systems
Zhu, Y., Wang, M., Yin, X., Zhang, J., Meijering, E. H. W., & Hu, J. (2022) · Sensors · doi:10.3390/s23010062 · Cited by 84
FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Rotstein, N., Bensaïd, D., Brody, S., Ganz, R., & Kimmel, R. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00559 · Cited by 66
Spatio–Spectral Representation Learning for Electroencephalographic Gait-Pattern Classification
Goh, S. K., Abbass, H. A., Tan, K. C., Al-Mamun, A., Thakor, N. V., Bezerianos, A., & Li, J. (2018) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2018.2864119 · Cited by 77
Defensive dropout for hardening deep neural networks under adversarial attacks
Wang, S., Wang, X., Zhao, P., Wen, W., Kaeli, D. R., Chin, P. S., & Lin, X. (2018) · doi:10.1145/3240765.3264699 · Cited by 82
Skomski, E., Lee, J., Kim, W., Chandan, V., Katipamula, S., & Hutchinson, B. (2020) · Energy and Buildings · doi:10.1016/j.enbuild.2020.110350 · Cited by 73
pix2code: Generating Code from a Graphical User Interface ScreenshotPreprint
Beltramelli, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07962 · Cited by 125
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech
Kamper, H., Settle, S., Shakhnarovich, G., & Livescu, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-502 · Cited by 64
Decoupled Novel Object Captioner
Wu, Y., Zhu, L., Jiang, L., & Yang, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240640 · Cited by 62
Automatic Arabic Image Captioning using RNN-LSTM-Based Language Model and CNN
Almuzaini, H. A., Tasniem, N., & Benhidour, H. (2018) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2018.090610 · Cited by 80
Integrating Part of Speech Guidance for Image Captioning
Zhang, J., Mei, K., Zheng, Y., & Fan, J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2976552 · Cited by 67
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
Huang, J.-H., Huck Yang, C.-H., Liu, F., Tian, M., Liu, Y.-C., Wu, T.-W., Lin, I.-H., Wang, K., Morikawa, H., Chang, H., et al. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00249 · Cited by 64
Double-Stream Position Learning Transformer Network for Image Captioning
Jiang, W., Zhou, W., & Hu, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3181490 · Cited by 70
Time Series Forecasting Using Sequence-to-Sequence Deep Learning Framework
Du, S., Li, T., & Horng, S. (2018) · International Symposium on Parallel Architectures, Algorithms and Programming · doi:10.1109/paap.2018.00037 · Cited by 67
Cui, W. H., Wang, F., He, X. J., Zhang, D., Xu, X., Meng, Y., Wang, Z., & Huang, J. (2019) · Remote Sensing · doi:10.3390/rs11091044 · Cited by 69
Encode, Review, and Decode: Reviewer Module for Caption Generation.Preprint
Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · Cited by 77
Reference Based LSTM for Image Captioning
Chen, M., Ding, G., Zhao, S., Chen, H., Liu, Q., & Han, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11198 · Cited by 63
Forecast of Solar Energy Production - A Deep Learning Approach
Zhang, R., Feng, M., Zhang, W., Lu, S., & Wang, F. (2018) · IEEE International Conference on Big Knowledge (ICBK) · doi:10.1109/icbk.2018.00018 · Cited by 78
Thomas, M. R., Martin, B. S., Kowarski, K. A., Gaudet, B. J., & Matwin, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46133-1_18 · Cited by 66
Cross-Domain Image Captioning via Cross-Modal Retrieval and Model Adaptation
Zhao, W., Wu, X., & Luo, J. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3042086 · Cited by 72
Automatic Image Annotation Based on Deep Learning Models: A Systematic Review and Future Challenges
Adnan, M. M., Rahim, M. S. M., Rehman, A. U., Mehmood, Z., Saba, T., & Naqvi, R. A. (2021) · IEEE Access · doi:10.1109/access.2021.3068897 · Cited by 81
The Role of Machine Learning in Digital Marketing
Ullal, M. S., Hawaldar, I. T., Soni, R., & Nadeem, M. (2021) · SAGE Open · doi:10.1177/21582440211050394 · Cited by 97
EI-CLIP: Entity-aware Interventional Contrastive Learning for E-commerce Cross-modal Retrieval
Ma, H., Zhao, H., Lin, Z., Kale, A., Wang, Z., Yu, T., Gu, J., Choudhary, S., & Xie, X. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01752 · Cited by 101
Semantic Invariant Multi-View Clustering With Fully Incomplete Information
Zeng, P., Yang, M., Lu, Y., Zhang, C., Hu, P., & Peng, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3332967 · Cited by 70
Computer Vision and Natural Language Processing: Recent Approaches in Multimedia and Robotics
Wiriyathammabhum, P., Summers-Stay, D. A., Fermuller, C. M., & Aloimonos, Y. (2016) · ACM Computing Surveys · doi:10.1145/3009906 · Cited by 86
A hierarchical and regional deep learning architecture for image description generation
Kinghorn, P., Zhang, L., & Shao, L. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.09.013 · Cited by 69
I2T2I: Learning text to image synthesis with textual data augmentation
Dong, H., Zhang, J., McIlwraith, D. G., & Guo, Y. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296635 · Cited by 76
Natural Language Guided Visual Relationship Detection
Liao, W., Rosenhahn, B., Ling, S., & Yang, M. Y. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00058 · Cited by 79
DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow.Preprint
Han, S., Pool, J., Narang, S., Mao, H., Tang, S., Elsen, E., Catanzaro, B., Tran, J. J., & Dally, W. J. (2016) · arXiv (Cornell University) · Cited by 64
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · arXiv · doi:10.48550/arxiv.1603.08507 · Cited by 64
Memory-Based Augmentation Network for Video Captioning
Jing, S., Zhang, H., Zeng, P., Gao, L., Song, J., & Shen, H. T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3295098 · Cited by 74
Common Subspace for Model and Similarity: Phrase Learning for Caption Generation from Images
Ushiku, Y., Yamaguchi, M., Mukuta, Y., & Harada, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.306 · Cited by 76
Object Referring in Videos with Language and Human Gaze
Vasudevan, A. B., Dai, D., & Gool, L. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00434 · Cited by 93
Fault Diagnosis of Asynchronous Motors Based on LSTM Neural Network
Xiao, D., Huang, Y., Zhang, X., Shi, H., Liu, C., & Li, Y. (2018) · Prognostics and System Health Management Conference · doi:10.1109/phm-chongqing.2018.00098 · Cited by 65
Su, P.-H., Vandyke, D., Gašić, M., Kim, D., Mrkšić, N., Wen, T.-H., & Young, S. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-456 · Cited by 63
Is image-based CAPTCHA secure against attacks based on machine learning? An experimental study
Alqahtani, F. H., & Alsulaiman, F. A. (2019) · Computers & Security · doi:10.1016/j.cose.2019.101635 · Cited by 76
Show, Recall, and Tell: Image Captioning with Recall Mechanism
Wang, L., Bai, Z., Zhang, Y., & Lu, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6898 · Cited by 57
Dual Attention on Pyramid Feature Maps for Image Captioning
Yu, L., Zhang, J., & Wu, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3072479 · Cited by 69
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.