Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 501–600 of 1,041
Recurrent Neural Networks: An Embedded Computing Perspective
Rezk, N. M., Purnaprajna, M., Nordström, T., & Ul-Abdin, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2982416 · Cited by 168
Fine-grained Image-text Matching by Cross-modal Hard Aligning Network
Pan, Z., Wu, F., & Zhang, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01847 · Cited by 142
Compressing Convolutional Neural Networks in the Frequency Domain
Chen, W., Wilson, J., Tyree, S., Weinberger, K. Q., & Chen, Y. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939839 · Cited by 127
Learning Fragment Self-Attention Embeddings for Image-Text Matching
Wu, Y., Wang, S., Song, G., & Huang, Q. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350940 · Cited by 160
Abbasi, A. A., Hussain, L., Awan, I. A., Abbasi, I., Majid, A., Nadeem, M. S. A., & Chaudhary, Q.-A. (2020) · Cognitive Neurodynamics · doi:10.1007/s11571-020-09587-5 · Cited by 154
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
Chen, W., Hu, H., Chen, X., Verga, P., & Cohen, W. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.375 · Cited by 125
Deep Compositional Question Answering with Neural Module NetworksPreprint
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2015) · arXiv (Cornell University) · Cited by 147
Reasoning Visual Dialogs With Structural and Partial Observations
Zheng, Z., Wang, W., Qi, S., & Zhu, S. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00683 · Cited by 146
Region-Object Relation-Aware Dense Captioning via Transformer
Shao, Z., Han, J., Marnerides, D., & Debattista, K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3152990 · Cited by 155
Learning Deep Representations of Fine-grained Visual Descriptions
Reed, S., Akata, Z., Schiele, B., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05395 · Cited by 130
Creativity: Generating Diverse Questions Using Variational Autoencoders
Jain, U., Zhang, Z., & Schwing, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.575 · Cited by 123
GLA: Global–Local Attention for Image Description
Li, L., Tang, S., Zhang, Y., Deng, L., & Tian, Q. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2751140 · Cited by 140
Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning
Wu, Q., Wang, P., Shen, C., Reid, I., & Hengel, A. V. D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00639 · Cited by 129
Learning Dual Semantic Relations With Graph Attention for Image-Text Matching
Wen, K., Gu, X., & Cheng, Q. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3030656 · Cited by 151
Deconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 126
Image Captioning with Deep Bidirectional LSTMs and Multi-Task Learning
Wang, C., Yang, H., & Meinel, C. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3115432 · Cited by 130
Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
What Does BERT with Vision Look At?
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.469 · Cited by 121
DeepArchitect: Automatically Designing and Training Deep Architectures
Negrinho, R. M. P., & Gordon, G. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08792 · Cited by 126
A Data Augmentation Approach for Sign-Language-To-Text Translation In-The-Wild
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.ldk.2021.36 · Cited by 128
Representation of Linguistic Form and Function in Recurrent Neural Networks
Kádár, Á., Chrupała, G., & Alishahi, A. (2017) · Computational Linguistics · doi:10.1162/coli_a_00300 · Cited by 128
Representations of language in a model of visually grounded speech signal
Chrupała, G., Gelderloos, L., & Alishahi, A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1057 · Cited by 146
Zhang, J., Du, J., & Dai, L.-R. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2844689 · Cited by 131
A long video caption generation algorithm for big video data retrieval
Ding, S., Qu, S., Xi, Y., & Wan, S. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.10.054 · Cited by 133
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128
Semantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 180
CPTR: Full Transformer Network for Image Captioning
Liu, W., Chen, S., Guo, L., Zhu, X., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10804 · Cited by 106
SemStyle: Learning to Generate Stylised Image Captions Using Unaligned Text
Mathews, A., Xie, L., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00896 · Cited by 125
Jin, W., Cheng, Y., Shen, Y., Chen, W., & Ren, X. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.197 · Cited by 116
Large-Scale Datasets for Going Deeper in Image Understanding
Wu, J., Zheng, H., Zhao, B., Li, Y., Yan, B., Liang, R., Wang, W., Zhou, S., Lin, G., Fu, Y., et al. (2019) · 2019 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00256 · Cited by 124
Early Action Prediction by Soft Regression
Hu, J.-F., Zheng, W.-S., Ma, L., Wang, G., Lai, J., & Zhang, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2863279 · Cited by 125
Text-based Person Search via Attribute-aided Matching
Aggarwal, S., Babu, R. V., & Chakraborty, A. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093640 · Cited by 129
Deep Hierarchical Encoder–Decoder Network for Image Captioning
Xiao, X., Wang, L., Ding, K., Xiang, S., & Pan, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2915033 · Cited by 141
Cross-modal Graph Matching Network for Image-text Retrieval
Cheng, Y., Zhu, X., Qian, J., Wen, F., & Liu, P. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3499027 · Cited by 154
Attentive Recurrent Social Recommendation
Sun, P., Wu, L., & Wang, M. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210023 · Cited by 134
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image ClassificationPreprint
Wang, Y., Lv, K., Huang, R., Song, S., Yang, L., & Huang, G. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05300 · Cited by 180
Pix2seq: A Language Modeling Framework for Object Detection
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 124
Answer-Type Prediction for Visual Question Answering
Kafle, K., & Kanan, C. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.538 · Cited by 122
Link the Head to the "Beak": Zero Shot Learning from Noisy Text Description at Part Precision
Elhoseiny, M., Zhu, Y., Zhang, H., & Elgammal, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.666 · Cited by 121
Chen, H., Zhang, H., Chen, P., Yi, J., & Hsieh, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1241 · Cited by 167
A survey of fractional calculus applications in artificial neural networks
Joshi, M. P., Bhosale, S. R., & Vyawahare, V. A. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10474-8 · Cited by 137
Anisotropic Diffusion Descriptors
Boscaini, D., Masci, J., Rodolà, E., Bronstein, M. M., & Cremers, D. (2016) · Computer Graphics Forum · doi:10.1111/cgf.12844 · Cited by 122
Detecting Unseen Visual Relations Using Analogies
Peyre, J., Šivic, J., Laptev, I., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00207 · Cited by 141
Combining Convolutional and Recurrent Neural Networks for Human Skin Detection
Zuo, H., Fan, H., Blasch, E., & Ling, H. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2654803 · Cited by 132
Plappert, M., Mandery, C., & Asfour, T. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.006 · Cited by 168
Query-Guided Regression Network with Context Policy for Phrase Grounding
Chen, K., Kovvuri, R., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.95 · Cited by 114
Deep Learning for Image-to-Text Generation: A Technical Overview
He, X., & Deng, L. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2741510 · Cited by 121
Conditional Image-Text Embedding Networks
Plummer, B. A., Kordas, P., Kiapour, M. H., Zheng, S., Piramuthu, R., & Lazebnik, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_16 · Cited by 135
Indicators and criteria of consciousness in animals and intelligent machines: an inside-out approach
Pennartz, C. M. A., Farisco, M., & Evers, K. (2019) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2019.00025 · Cited by 121
Saliency-Guided Attention Network for Image-Sentence Matching
Ji, Z., Wang, H., Han, J., & Pang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00585 · Cited by 133
The Long-Short Story of Movie Description
Rohrbach, A., Rohrbach, M., & Schiele, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24947-6_17 · Cited by 126
Learning Contextual Dependence With Convolutional Hierarchical Recurrent Neural Networks
Zuo, Z. C., Shuai, B., Wang, G., Liu, X., Wang, X., Wang, B., & Chen, Y. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2548241 · Cited by 130
Sharma, A., Liu, X., & Yang, X. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.05.019 · Cited by 128
Learning Structured Inference Neural Networks with Label Relations
Hu, H., Zhou, G.-T., Deng, Z., Liao, Z., & Mori, G. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.323 · Cited by 142
Wang, Q., Ripamonti, N., & Hesthaven, J. S. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.109402 · Cited by 123
Neural Machine Translation and Sequence-to-sequence Models: A TutorialPreprint
Neubig, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01619 · Cited by 217
Weakly-Supervised Visual Grounding of Phrases with Linguistic Structures
Xiao, F., Sigal, L., & Lee, Y. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.558 · Cited by 140
Automatic Generation of Medical Imaging Diagnostic Report with Hierarchical Recurrent Neural Network
Yin, C., Qian, B., Wei, J., Li, X., Zhang, X., Li, Y., & Zheng, Q. (2019) · 2019 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2019.00083 · Cited by 98
Learning Semantic-Aligned Feature Representation for Text-Based Person Search
Li, S., Cao, M., & Zhang, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746846 · Cited by 121
Light-Weight RefineNet for Real-Time Semantic SegmentationPreprint
Nekrasov, V., Shen, C., & Reid, I. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03272 · Cited by 160
Gan, C., Li, Y., Li, H., Sun, C., & Gong, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.201 · Cited by 116
Multimodal Prototypical Networks for Few-shot Learning
Pahde, F., Puscas, M., Klein, T., & Nabi, M. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00269 · Cited by 110
Two-Stage Attention Over LSTM With Bayesian Optimization for Day-Ahead Solar Power Forecasting
Aslam, M., Lee, S., Khang, S.-H., & Hong, S. (2021) · IEEE Access · doi:10.1109/access.2021.3100105 · Cited by 134
Step-Wise Hierarchical Alignment Network for Image-Text Matching
Ji, Z., Chen, K., & Wang, H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/106 · Cited by 120
Multimodal Federated Learning on IoT Data
Zhao, Y., Barnaghi, P. M., & Haddadi, H. (2022) · IEEE/ACM International Conference on Internet-of-Things Design and Implementation (IoTDI) · doi:10.1109/iotdi54339.2022.00011 · Cited by 170
Deep learning and the global workspace theory
VanRullen, R., & Kanai, R. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.04.005 · Cited by 128
Automatic Concept Discovery from Parallel Text and Visual Corpora
Sun, C., Gan, C., & Nevatia, R. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.298 · Cited by 111
Enhancing Video Summarization via Vision-Language Embedding
Plummer, B. A., Brown, M., & Lazebnik, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.118 · Cited by 113
Federated Learning for Vision-and-Language Grounding Problems
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6824 · Cited by 147
Reflective Decoding Network for Image Captioning
Ke, L., Pei, W., Li, R., Shen, X., & Tai, Y.-W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00898 · Cited by 109
Contextual Object Detection with Multimodal Large Language Models
Zang, Y., Li, W., Han, J., Zhou, K., & Loy, C. C. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02214-4 · Cited by 240
Unrestricted Adversarial Examples via Semantic ManipulationPreprint
Bhattad, A., Chong, M. J., Liang, K., Li, B., & Forsyth, D. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.06347 · Cited by 192
Context-Aware Captions from Context-Agnostic Supervision
Vedantam, R., Bengio, S., Murphy, K., Parikh, D., & Chechik, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.120 · Cited by 113
Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
Wang, Y., Lin, Z., Shen, X., Cohen, S., & Cottrell, G. W. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.780 · Cited by 112
Fully Convolutional Scene Graph Generation
Liu, H., Yan, N., Mortazavi, M., & Bhanu, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01138 · Cited by 149
Women also Snowboard: Overcoming Bias in Captioning Models
Burns, K., Hendricks, L. A., Saenko, K., Darrell, T. J., & Rohrbach, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09797 · Cited by 100
Evaluating the descriptive power of Instagram hashtags
Giannoulakis, S., & Tsapatsoulis, N. (2016) · Journal of Innovation in Digital Ecosystems · doi:10.1016/j.jides.2016.10.001 · Cited by 130
James, C. D., Aimone, J. B., Miner, N. E., Vineyard, C. M., Rothganger, F. H., Carlson, K. D., Mulder, S. A., Draelos, T. J., Faust, A., Marinella, M. J., et al. (2017) · Biologically Inspired Cognitive Architectures · doi:10.1016/j.bica.2016.11.002 · Cited by 153
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion
Yao, K., & Zweig, G. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-134 · Cited by 169
Where to put the image in an image caption generator
Tanti, M., Albert Gatt, A., & Camilleri, K. P. (2017) · OAR@UM (University of Malta) · Cited by 112
CLEVR-Ref+: Diagnosing Visual Reasoning With Referring Expressions
Liu, R., Liu, C., Bai, Y., & Yuille, A. L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00431 · Cited by 109
Deep Adversarial Graph Attention Convolution Network for Text-Based Person Search
Liu, J., Zha, Z.-J., Hong, R., Wang, M., & Zhang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350991 · Cited by 126
Dave, S., Baghdadi, R., Nowatzki, T., Avancha, S., Shrivastava, A., & Li, B. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3098483 · Cited by 111
Rich Image Captioning in the Wild
Tran, K., He, X., Zhang, L., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.61 · Cited by 103
Table Cell Search for Question Answering
Sun, H., Ma, H., He, X., Yih, W.-T., Su, Y., & Yan, X. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883080 · Cited by 107
Learning Bidirectional Temporal Cues for Video-Based Person Re-Identification
Zhang, W., Yu, X., & He, X. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2718188 · Cited by 114
Feature Space Transfer for Data Augmentation
Liu, B., Wang, X., Dixit, M., Kwitt, R., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00947 · Cited by 101
Li, B., Wu, G., He, Y., Fan, M., & Pedrycz, W. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105677 · Cited by 146
LapsCore: Language-guided Person Search via Color Reasoning
Wu, Y., Yan, Z., Han, X., Li, G., Zou, C., & Cui, S. R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00165 · Cited by 119
Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach
Wang, Z., Han, H., Wang, L., Li, X., & Zhou, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3171661 · Cited by 108
Medical image captioning via generative pretrained transformers
Selivanov, A., Rogov, O. Y., Chesakov, D., Shelmanov, A., Fedulova, I., & Dylov, D. V. (2023) · Scientific Reports · doi:10.1038/s41598-023-31223-5 · Cited by 107
Multi-cue Zero-Shot Learning with Strong Supervision
Akata, Z., Malinowski, M., Fritz, M., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.14 · Cited by 104
Automatic Generation of Text Descriptive Comments for Code Blocks
Liang, Y., & Zhu, K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11963 · Cited by 106
Dual Global Enhanced Transformer for image captioning
Xian, T., Li, Z., Zhang, C., & Ma, H. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.01.011 · Cited by 119
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
Che, Z., Purushotham, S., Khemani, R. G., & Liu, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.03542 · Cited by 105
Trellis Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06682 · Cited by 90
Semantic Regularisation for Recurrent Image Annotation
Liu, F., Xiang, T., Hospedales, T. M., Yang, W., & Sun, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.443 · Cited by 119
Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00419 · Cited by 120
Visual to Text: Survey of Image and Video Captioning
Li, S., Tao, Z., Li, K., & Fu, Y. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2892755 · Cited by 103
Schwartz, I., Yu, S., Hazan, T., & Schwing, A. G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00214 · Cited by 106
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.