Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 601–700 of 1,041
Automatic Image Captioning Based on ResNet50 and LSTM with Soft Attention
Chu, Y., Xiao, Y., Yu, L., Sergei, M., & Wang, Z. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8909458 · Cited by 137
You, Q., Cao, L., Jin, H., & Luo, J. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964288 · Cited by 129
Cross on Cross Attention: Deep Fusion Transformer for Image Captioning
Zhang, J., Xie, Y., Ding, W., & Wang, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3243725 · Cited by 116
Towards Unsupervised Image Captioning With Shared Multimodal Embeddings
Laina, I., Rupprecht, C., & Navab, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00751 · Cited by 111
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering
Malinowski, M., Rohrbach, M., & Fritz, M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1038-2 · Cited by 105
Wu, J., Tenenbaum, J. B., & Kohli, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.744 · Cited by 104
Image Understanding using vision and reasoning through Scene Description Graph
Aditya, S., Yang, Y., Baral, C. R., Aloimonos, Y., & Fermuller, C. M. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.12.004 · Cited by 123
Context-Aware Visual Policy Network for Sequence-Level Image Captioning
Liu, D., Zha, Z.-J., Zhang, H., Zhang, Y., & Wu, F. H. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240632 · Cited by 119
Unconstrained Salient Object Detection via Proposal Subset Optimization
Zhang, J., Sclaroff, S., Lin, Z., Shen, X., Price, B. L., & Měch, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.618 · Cited by 115
Cross-media analysis and reasoning: advances and directions
Peng, Y., Zhu, W., Zhao, Y., Xu, C., Huang, Q., Lu, H., Zheng, Q., Huang, T., & Gao, W. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601787 · Cited by 108
A region-based image caption generator with refined descriptions
Kinghorn, P., Zhang, L., & Shao, L. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.07.014 · Cited by 119
Learning Visual Question Answering by Bootstrapping Hard Attention
Malinowski, M., Doersch, C., Santoro, A., & Battaglia, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_1 · Cited by 118
Bi-Directional Spatial-Semantic Attention Networks for Image-Text Matching
Huang, F., Zhang, X., Zhao, Z., & Li, Z. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2882225 · Cited by 107
RODNet: Radar Object Detection using Cross-Modal Supervision
Wang, Y., Jiang, Z., Gao, X., Hwang, J., Xing, G., & Liu, H. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00055 · Cited by 150
Zhao, H., Puig, X., Zhou, B., Fidler, S., & Torralba, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.221 · Cited by 99
Query-adaptive Video Summarization via Quality-aware Relevance Estimation
Vasudevan, A. B., Gygli, M., Volokitin, A., & Gool, L. V. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123297 · Cited by 116
Improving Image Captioning with Conditional Generative Adversarial Nets
Chen, C., Mu, S., Xiao, W., Ye, Z., Wu, L., & Ju, Q. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018142 · Cited by 117
SAT: 2D Semantics Assisted Training for 3D Visual Grounding
Yang, Z., Zhang, S., Wang, L., & Luo, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00187 · Cited by 184
Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions
Wang, T., Li, F., Zhu, L., Li, J., Zhang, Z., & Shen, H. T. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3525147 · Cited by 241
Learning Word-Like Units from Joint Audio-Visual Analysis
Harwath, D. F., & Glass, J. R. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1047 · Cited by 114
Zhong, G., Ling, X., & Wang, L. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1255 · Cited by 119
Captioning Transformer with Stacked Attention Modules
Zhu, X., Li, L., Liu, J., Peng, H., & Niu, X. (2018) · Applied Sciences · doi:10.3390/app8050739 · Cited by 102
Image Captioning With End-to-End Attribute Detection and Subsequent Attributes Prediction
Huang, Y., Chen, J., Ouyang, W., Wan, W., & Xue, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2969330 · Cited by 111
Deep Multimodal Neural Architecture Search
Zhou, Y., Cui, Y., Yu, J., Wang, M., Tao, D., & Tian, Q. C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413977 · Cited by 122
See, Hear, and Read: Deep Aligned RepresentationsPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00932 · Cited by 155
The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
Iyyer, M., Manjunatha, V., Guha, A., Vyas, Y., Boyd-Graber, J. L., Daumé, H., & Davis, L. S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.686 · Cited by 139
Topic-Oriented Image Captioning Based on Order-Embedding
Yu, N., Hu, X., Song, B., Yang, J., & Zhang, J. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2889922 · Cited by 106
Face attribute prediction using off-the-shelf CNN features
Zhong, Y., Sullivan, J., & Li, H. (2016) · International Conference on Biometrics · doi:10.1109/icb.2016.7550092 · Cited by 116
Learning to Collocate Neural Modules for Image Captioning
Yang, X., Zhang, H., & Cai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00435 · Cited by 96
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100
Using human brain activity to guide machine learning
Fong, R., Scheirer, W. J., & Cox, D. (2018) · Scientific Reports · doi:10.1038/s41598-018-23618-6 · Cited by 124
A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning
Huynh, D., & Elhamifar, E. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00880 · Cited by 130
Stochastic Multiple Choice Learning for Training Diverse Deep EnsemblesPreprint
Lee, S., Purushwalkam, S., Cogswell, M., Ranjan, V., Crandall, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07839 · Cited by 211
Grounding Distributional Semantics in the Visual World
Baroni, M. (2016) · Language and Linguistics Compass · doi:10.1111/lnc3.12170 · Cited by 99
Ghodrati, M., Khaligh‐Razavi, S., & Lehky, S. R. (2017) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2017.06.002 · Cited by 120
Interpretable Predictions of Clinical Outcomes with An Attention-based Recurrent Neural Network
Sha, Y., & Wang, M. D. (2017) · Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics · doi:10.1145/3107411.3107445 · Cited by 97
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
Cornia, M., Baraldi, L., Serra, G., & Cucchiara, R. (2018) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3177745 · Cited by 124
Learning to Act Properly: Predicting and Explaining Affordances from Images
Chuang, C.-Y., Li, J., Torralba, A., & Fidler, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00108 · Cited by 98
Grounded Situation Recognition
Pratt, S. M., Yatskar, M., Weihs, L., Farhadi, A., & Kembhavi, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_19 · Cited by 148
Image Captioning: A Comprehensive Survey
Sharma, H., Agrahari, M., Singh, S. K., Firoj, M., & Mishra, R. K. (2020) · International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC) · doi:10.1109/parc49193.2020.236619 · Cited by 113
A holistic overview of deep learning approach in medical imaging
Yousef, R., Gupta, G., Yousef, N., & Khari, M. (2022) · Multimedia Systems · doi:10.1007/s00530-021-00884-5 · Cited by 127
Exploiting Unlabeled Data with Vision and Language Models for Object Detection
Zhao, S., Zhang, Z.-X., Schulter, S., Zhao, L., Kumar, B. G. V., Stathopoulos, A., Chandraker, M. K., & Metaxas, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20077-9_10 · Cited by 164
Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
Kuo, C.-W., & Kira, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01744 · Cited by 121
Generating News Headlines with Recurrent Neural NetworksPreprint
Lopyrev, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.01712 · Cited by 136
Audio visual speech recognition with multimodal recurrent neural networks
Feng, W., Guan, N., Li, Y., Zhang, X., & Luo, Z. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965918 · Cited by 129
Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation
Huang, Q., Gan, Z., Celikyilmaz, A., Wu, D., Wang, J., & He, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018465 · Cited by 98
Grow and Prune Compact, Fast, and Accurate LSTMs
Dai, X., Yin, H., & Jha, N. K. (2019) · IEEE Transactions on Computers · doi:10.1109/tc.2019.2954495 · Cited by 114
Cross-media Structured Common Space for Multimedia Event Extraction
Li, M., Zareian, A., Zeng, Q., Whitehead, S., Lu, D., Ji, H., & Chang, S.-F. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.230 · Cited by 91
A survey on visual navigation for artificial agents with deep reinforcement learning
Zeng, F., Wang, C., & Ge, S. S. (2020) · IEEE Access · doi:10.1109/access.2020.3011438 · Cited by 98
Relation extraction from clinical texts using domain invariant convolutional neural network
Sahu, S. K., Anand, A., Oruganty, K., & Gattu, M. (2016) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/w16-2928 · Cited by 117
Bottom-Up and Top-Down Reasoning with Hierarchical Rectified Gaussians
Hu, P., & Ramanan, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.604 · Cited by 94
Yu, J., Zhang, W., Lu, Y., Qin, Z., Hu, Y., Tan, J., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972830 · Cited by 97
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos
Chen, S., Jiang, W., Liu, W., & Jiang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_20 · Cited by 126
Combining Weight Pruning and Knowledge Distillation For CNN Compression
Aghli, N., & Ribeiro, E. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00356 · Cited by 144
The Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis
Barraco, M., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00512 · Cited by 137
Jin, J., Fu, K., Cui, R., Sha, F., & Zhang, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06272 · Cited by 97
Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2020) · arXiv (Cornell University) · doi:10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m · Cited by 97
Deep learning: from speech recognition to language and multimodal processing
Deng, L. (2016) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2015.22 · Cited by 110
Natural Language Statistical Features of LSTM-Generated Texts
Lippi, M., Montemurro, M. A., Esposti, M. D., & Cristadoro, G. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2890970 · Cited by 105
Multi-modal Circulant Fusion for Video-to-Language and Backward
Wu, A., & Han, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/143 · Cited by 99
Differential Angular Imaging for Material Recognition
Xue, J., Zhang, H., Dana, K., & Nishino, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.734 · Cited by 107
Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
Chen, Z., Ma, L., Luo, W., & Wong, K.-Y. K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1183 · Cited by 135
Convolutional Neural Network Hyper-Parameters Optimization based on Genetic Algorithms
Loussaief, S., & Abdelkrim, A. (2018) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2018.091031 · Cited by 110
Look, Listen and Learn — A Multimodal LSTM for Speaker Identification
Ren, J., Hu, Y., Tai, Y.-W., Wang, C., Xu, L., Sun, W., & Yan, Q. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10471 · Cited by 91
Towards Real-Time Object Detection on Embedded Systems
Mao, H., Yao, S., Tang, T., Li, B., Yao, J., & Wang, Y. (2018) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2016.2593643 · Cited by 94
Multimodal Pivots for Image Caption Translation
Hitschler, J., Schamoni, S., & Riezler, S. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1227 · Cited by 88
BreakingNews: Article Annotation by Image and Text Processing
Ramisa, A., Yan, F., Moreno-Noguer, F., & Mikolajczyk, K. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2721945 · Cited by 93
Using synthetic data to train neural networks is model-based reasoning
Le, T., Baydin, A. G., Zinkov, R., & Wood, F. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966298 · Cited by 127
Fluency-Guided Cross-Lingual Image Captioning
Lan, W., Li, X., & Dong, J. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123366 · Cited by 92
Zamir, A., Wu, T.-L., Sun, L., Shen, W., Shi, B. E., Malik, J., & Savarese, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · Cited by 118
Knowledge Aided Consistency for Weakly Supervised Phrase Grounding
Chen, K., Gao, J., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00425 · Cited by 107
Localizing Natural Language in Videos
Chen, J., Ma, L., Chen, X., Jie, Z., & Luo, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018175 · Cited by 115
Scene graph captioner: Image captioning based on structural visual representation
Xu, N., Liu, A.-A., Liu, J., Nie, W., & Su, Y. (2018) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2018.12.027 · Cited by 104
Vaddireddy, H., Rasheed, A., Staples, A., & San, O. (2020) · Physics of Fluids · doi:10.1063/1.5136351 · Cited by 114
Orderless Recurrent Models for Multi-Label Classification
Yazici, V. O., González-García, A., Arnau Ramisa, A., Twardowski, B., & van de Weijer, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01345 · Cited by 145
INGRESS: Interactive visual grounding of referring expressions
Shridhar, M., Mittal, D., & Hsu, D. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919897133 · Cited by 113
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299
Gordo, A., & Larlus, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.560 · Cited by 107
Text2Scene: Generating Compositional Scenes From Textual Descriptions
Tan, F., Feng, S., & Ordonez, V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00687 · Cited by 93
Object Detection with Transformers: A Review
Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2025) · Sensors · doi:10.3390/s25196025 · Cited by 93
Survey on visual sentiment analysis
Ortis, A., Farinella, G. M., & Battiato, S. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.1270 · Cited by 103
A Self-boosting Framework for Automated Radiographic Report Generation
Wang, Z., Zhou, L., Wang, L., & Li, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00246 · Cited by 83
Lebret, R., Pinheiro, P. O., & Collobert, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.03671 · Cited by 45
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
Datta, S., Sikka, K., Roy, A., Ahuja, K., Parikh, D., & Divakaran, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00269 · Cited by 93
Where to put the image in an image caption generator
TANTI, M., Albert Gatt, A., & Camilleri, K. P. (2018) · Natural Language Engineering · doi:10.1017/s1351324918000098 · Cited by 150
Unpaired Image Captioning by Language Pivoting
Gu, J., Joty, S., Cai, J., & Wang, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_31 · Cited by 93
Visual question answering model based on visual relationship detection
Xi, Y., Zhang, Y., Ding, S., & Wan, S. (2019) · Signal Processing Image Communication · doi:10.1016/j.image.2019.115648 · Cited by 99
Transform and Tell: Entity-Aware News Image Captioning
Tran, A., Mathews, A., & Xie, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01305 · Cited by 87
Boroumand, A., Ghose, S., Akin, B., Narayanaswami, R., Junior, G. F. D. O., Ma, X., Shiu, E., & Mutlu, O. (2021) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact52795.2021.00019 · Cited by 153
Unsupervised Semantic Parsing of Video Collections
Sener, O., Zamir, A. R., Savarese, S., & Saxena, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.509 · Cited by 89
An accident diagnosis algorithm using long short-term memory
Yang, J., & Kim, J.-H. (2018) · Nuclear Engineering and Technology · doi:10.1016/j.net.2018.03.010 · Cited by 92
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 47
A survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90
Data science as machinic neoplatonism
McQuillan, D. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0273-3 · Cited by 90
Active garment recognition and target grasping point detection using deep learning
Corona, E., Alenyà, G., Gabas, A., & Torras, C. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.09.042 · Cited by 90
"You Know What to Do": Proactive Detection of YouTube Videos Targeted by Coordinated Hate Attacks
Mariconti, E., Suárez‐Tangil, G., Blackburn, J., Cristofaro, E. D., Kourtellis, N., Leontiadis, I., Serrano, J. L., & Stringhini, G. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359309 · Cited by 109
Neural representation for object recognition in inferotemporal cortex
Lehky, S. R., & Tanaka, K. (2016) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2015.12.001 · Cited by 101
Social Media Image Analysis for Public Health
Garimella, V. R. K., Alfayad, A., & Weber, I. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858234 · Cited by 89
CASCADE: Connecting RRAMs to Extend Analog Dataflow In An End-To-End In-Memory Processing Paradigm
Chou, T., Tang, W., Botimer, J., & Zhang, Z. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358328 · Cited by 92
Denoising arterial spin labeling perfusion MRI with deep machine learning
Xie, D., Li, Y., Yang, H., Li, B., Wang, T., Zhou, F., Zhang, L., & Wang, Z. (2020) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2020.01.005 · Cited by 94
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.