Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 1,001–1,042 of 1,042
Zero-shot learning and its applications from autonomous vehicles to COVID-19 diagnosis: A review
Rezaei, M., & Shahidi, M. (2020) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2020.100005 · Cited by 72
Audio Caption: Listen and Tell
Wu, M., Dinkel, H., & Yu, K. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682377 · Cited by 52
RUBi: Reducing Unimodal Biases for Visual Question Answering
(2019) · HAL · Cited by 50
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Image captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49
Mitigating Gender Bias in Captioning Systems
Tang, R., Du, M., Li, Y., Liu, Z., Zou, N., & Hu, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449950 · Cited by 50
(2023) · doi:10.1109/iros55552.2023.10341379 · Cited by 48
Jiang, J., Lu, W., Chen, J., Lin, Q., Zhao, P., Kang, Y., Zhang, H., Xiong, Y., Gao, F., Xu, Z., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417054 · Cited by 56
Seeing with Humans: Gaze-Assisted Neural Image Captioning
Sugano, Y., & Bulling, A. (2016) · arXiv · doi:10.48550/arxiv.1608.05203 · Cited by 47
AutoAD: Movie Description in Context
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01815 · Cited by 43
Visual Semantic Information Pursuit: A Survey
Liu, D., Bober, M., & Kittler, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2950025 · Cited by 39
Computer Vision and Conflicting Values: Describing People with Automated Alt Text
Hanley, M., Barocas, S., Levy, K., Azenkot, S., & Nissenbaum, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462620 · Cited by 38
GilBERT: Generative Vision-Language Pre-Training for Image-Text Retrieval
Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., & Chu, W. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462838 · Cited by 37
COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation
Wen, K., Xia, J., Huang, Y., Li, L., Xu, J., & Shao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00221 · Cited by 37
Video Object Grounding Using Semantic Roles in Language Description
Sadhu, A., Chen, K., & Nevatia, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01043 · Cited by 36
Guo, W., Huang, H., Kong, X., & He, R. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3351053 · Cited by 36
(2016) · arXiv (Cornell University) · Cited by 29
Visual Question Generation: The State of the Art
Patil, C., & Patwardhan, M. (2021) · ACM Computing Surveys · doi:10.1145/3383465 · Cited by 25
Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization
Gidaris, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.90 · Cited by 23
Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Ji, Z., Shaikh, M. A., Moukheiber, D., Srihari, S. N., Peng, Y., & Gao, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87589-3_12 · Cited by 21
Xu, C., Yang, M., Ao, X., Shen, Y., Xu, R., & Tian, J. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106730 · Cited by 21
Abdulgalil, H. D., & Basir, O. A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100159 · Cited by 20
Automatic Medical Image Report Generation with Multi-view and Multi-modal Attention Mechanism
Yang, S., Niu, J., Wu, J., & Liu, X. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60248-2_48 · Cited by 17
Automatically Dismantling Online Dating Fraud
Suarez-Tangil, G., Edwards, M., Peersman, C., Stringhini, G., Rashid, A., & Whitty, M. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2019.2930479 · Cited by 17
ABYOLOv4: improved YOLOv4 human object detection based on enhanced multi-scale feature fusion
Li, R., Zeng, X., Yang, S., Li, Q., Yan, A., & Li, D. (2024) · EURASIP Journal on Advances in Signal Processing · doi:10.1186/s13634-023-01105-z · Cited by 15
Prediction and Privacy Scheme for Traffic Flow Estimation on the Highway Road Network
Akallouch, M., Akallouch, O., Fardousse, K., Bouhoute, A., & Berrada, I. (2022) · Information · doi:10.3390/info13080381 · Cited by 13
Guiding Visual Question Generation
Vedd, N., Wang, Z., Rei, M., Miao, Y., & Specia, L. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.118 · Cited by 12
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149
Im, D., Kim, T., Kim, B.-S., Park, J.-H., Kim, J.-G., & Park, Y.-J. (2025) · Tribology International · doi:10.1016/j.triboint.2025.110597 · Cited by 10
What BERT Sees: Cross-Modal Transfer for Visual Question Generation
Scialom, T., Bordes, P., Dray, P.-A., Staiano, J., & Gallinari, P. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.39 · Cited by 9
Vision to Language: Methods, Metrics and Datasets
Sharif, N., Nadeem, U., Shah, S. A. A., Bennamoun, M., & Liu, W. (2020) · Learning and Analytics in Intelligent Systems · doi:10.1007/978-3-030-49724-8_2 · Cited by 9
AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation
Yang, J.-H., Lassance, C., Sampaio De Rezende, R., Srinivasan, K., Redi, M., Clinchant, S., & Lin, J. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591903 · Cited by 9
Visual Grounding of Whole Radiology Reports for 3D CT Images
Ichinose, A., Hatsutani, T., Nakamura, K., Kitamura, Y., Iizuka, S., Simo-Serra, E., Kido, S., & Tomiyama, N. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43904-9_59 · Cited by 6
An Image Captioning Model Based on Bidirectional Depth Residuals and its Application
Zhou, Z., Xu, L., Wang, C., Xie, W., Wang, S., Ge, S., & Zhang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3057091 · Cited by 6
Semi-supervised multimodal representation learning through a global workspace
Devillers, B., Maytié, L., & VanRullen, R. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3416701 · Cited by 5
Exploring Uncertainty Measures for Image-Caption Embedding-and-Retrieval Task
Hama, K., Matsubara, T., Uehara, K., & Cai, J. (2019) · arXiv · doi:10.48550/arxiv.1904.08504 · Cited by 5
Peri, D., Sah, S., & Ptucha, R. (2019) · 2019 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2019.8802922 · Cited by 4
Probabilistic framework for solving visual dialog
Patro, B. N., Anupriy, & Namboodiri, V. P. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107586 · Cited by 4
Fusion or defusion? Flexible vision-and-language pre-training
Sun, R., Li, Z., Ding, Y., Wang, Q., Wang, J., Zheng, H., Wu, W., & Xian, Y. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.316 · Cited by 2
Sutter, T. M., Daunhawer, I., & Vogt, J. E. (2021) · arXiv · doi:10.48550/arxiv.2105.02470 · Cited by 41
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., & Socher, R. (2015) · arXiv · doi:10.48550/arxiv.1506.07285 · Cited by 1,020
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.