The DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Show and tell: A neural image caption generator
Abstract
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They coupled a convolutional neural network for encoding of the image, with a recurrent neural network - specifically a long-short term memory net (LSTM) (Hochreiter and Schmidhuber (1997)) - for textual encoding.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Vinyals et al. (2015)
In the citing paper
“They used human-generated sentences as ground truth for training, and used the bilingual evaluation understudy (BLEU) metric for evaluation.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Vinyals et al. (2015)
In the citing paper
“Vinyals et al. (2015) provided textual explanation for images using an end-to-end image captioning framework.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Vinyals et al. (2015)
In the citing paper
“Similarly, an image and a natural language caption can capture complimentary but converging information about a scene [32, 35].”
From Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018) · cites this paper as [32]
In the citing paper
“Dependence on complex datasets including images and textual annotations”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [43]
In the citing paper
“Semantic relationship between images and textual descriptions is intuitive, even for nonexperts”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [43]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Flagged first, then by citations
3 of the 40 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- kuznetsova, 2014, Treetalk: Composition and compression of trees for image descriptions, ACL, 2
Cited by
Sorted by citations
Items 501–600 of 1,039
Language-Based Image Editing with Recurrent Attentive Models
Chen, J., Shen, Y., Gao, J., Liu, J., & Liu, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00909 · Cited by 141
MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses
Hasan, I., Setti, F., Tsesmelis, T., Bue, A. D., Galasso, F., & Cristani, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00635 · Cited by 145
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
Ma, X., Zhou, C., Li, X., Neubig, G., & Hovy, E. H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1437 · Cited by 158
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., & Gao, J. (2024) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000110 · Cited by 331
Parallel Attention: A Unified Framework for Visual Object Discovery Through Dialogs and Queries
Zhuang, B., Wu, Q., Shen, C., Reid, I. D., & van den Hengel, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00447 · Cited by 146
Gist: Efficient Data Encoding for Deep Neural Network Training
Jain, A., Phanishayee, A., Mars, J., Tang, L., & Pekhimenko, G. G. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00070 · Cited by 151
Neural Open Information Extraction
Cui, L., Wei, F., & Zhou, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2065 · Cited by 158
CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback
Lee, S., Kim, D.-W., & Han, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00086 · Cited by 144
KiUT: Knowledge-injected U-Transformer for Radiology Report Generation
Huang, Z., Zhang, X., & Zhang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01897 · Cited by 162
“Despite the remarkable performance, these models restrained themselves in the methodology of image caption [6, 7, 36, 39, 43], and suffer from such data biases: 1) the normal cases dominate the dataset over the abnormal cases; 2) the descriptions of normal regions dominate the entire report.”
Hybrid Neural Networks for Learning the Trend in Time Series
Lin, T., Guo, T., & Aberer, K. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/316 · Cited by 145
Learning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00608 · Cited by 148
Explainable skin lesion diagnosis using taxonomies
Barata, C., Celebi, M. E., & Marques, J. S. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107413 · Cited by 153
Smart Reply: Automated Response Suggestion for EmailPreprint
Kannan, A., Kurach, K., Ravi, S., Kaufmann, T., Tomkins, A., Miklos, B., Corrado, G. S., Lukacs, L., Ganea, M., Young, P., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04870 · Cited by 148
Exploring Visual Relationship for Image CaptioningPreprint
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07041 · Cited by 141
Hierarchical Attention Network for Image Captioning
Wang, W., Chen, Z., & Hu, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018957 · Cited by 140
Image Captioning Through Image Transformer
He, S., Liao, W., Tavakoli, H. R., Yang, M. Y., Rosenhahn, B., & Pugeault, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_10 · Cited by 147
Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model
Yang, R., Mentzer, F., Gool, L. V., & Timofte, R. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3043590 · Cited by 148
Injecting Semantic Concepts into End-to-End Image Captioning
Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01748 · Cited by 146
“Recent studies have witnessed its great development which are primarily reflected in the aspects of more advanced cross-modal fusion architectures [11, 45, 50, 54, 57, 63, 65, 67, 70]; more expressive object-centric features [4, 69] & tags [16, 23, 35, 58] obtained from a pre-trained object detection model; or learnIMG”
BackgroundMethodI am Robot: (Deep) Learning to Break Semantic Image CAPTCHAs
Sivakorn, S., Polakis, I., & Keromytis, A. D. (2016) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp.2016.37 · Cited by 139
Situation Recognition with Graph Neural Networks
Li, R., Tapaswi, M., Liao, R., Jia, J., Urtasun, R., & Fidler, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.448 · Cited by 143
Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
Ye, J., Wang, L., Li, G., Chen, D., Zhe, S., Chu, X., & Xu, Z. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00977 · Cited by 147
Comprehensive Image Captioning via Scene Graph Decomposition
Zhong, Y., Wang, L., Chen, J., Yu, D., & Li, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_13 · Cited by 140
Competence-based Multimodal Curriculum Learning for Medical Report Generation
Liu, F., Ge, S., & Wu, X. (2021) · doi:10.18653/v1/2021.acl-long.234 · Cited by 153
“Such an encoder-decoder framework have achieved great success in advancing the state-of-the-arts (Vinyals et al., 2015; Lu et al., 2017; Xu et al., 2015; Cornia et al., 2020; Pan et al., 2020; Liu et al., 2020a).”
BackgroundMethodInfluentialTowards local visual modeling for image captioning
Ma, Y., Ji, J., Sun, X., Zhou, Y., & Ji, R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109420 · Cited by 139
“Specifically, [15] uses Convolutional Neural Network (CNN) to encode images and adopts Long Short-Term Memory (LSTM) as a decoder to generate captions.”
MethodImproving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149
Extending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 139
Hefron, R. G., Borghetti, B. J., Christensen, J. C., & Kabban, C. M. S. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.05.020 · Cited by 143
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04559-1 · Cited by 158
Transformers in computational visual media: A survey
Xu, Y., Wei, H., Lin, M., Deng, Y., Sheng, K., Zhang, M., Tang, F., Dong, W., Huang, F., & Xu, C. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0247-3 · Cited by 144
Comprehending and Ordering Semantics for Image Captioning
Li, Y., Pan, Y., Yao, T., & Mei, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01746 · Cited by 139
“In analogy to the RNN-based sequence modeling in machine translation, the earlier attempts [21, 33] directly employ the basic RNNbased encoder-decoder scheme for the task of image captioning, by encoding visual content with CNN and decoding output description with RNN.”
BackgroundMethodSequence to Sequence -- Video to TextPreprint
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R. J., Darrell, T., & Saenko, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00487 · Cited by 217
Referring Expression Generation and Comprehension via Attributes
Liu, J., Wang, L., & Yang, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.520 · Cited by 134
Improving Text-Based Person Search by Spatial Matching and Adaptive Threshold
Chen, T., Xu, C., & Luo, J. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00208 · Cited by 137
Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
Liu, X., Li, H., Shao, J., Chen, D., & Wang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_21 · Cited by 137
PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
Zhang, H., Kyaw, Z., Yu, J., & Chang, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.454 · Cited by 136
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
Chen, D. Z., Gholami, A., Nießner, M., & Chang, A. X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00321 · Cited by 157
Stangl, A., Morris, M. R., & Gurari, D. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376404 · Cited by 139
Good News, Everyone! Context Driven Entity-Aware Captioning for News Images
Biten, A. F., Gomez, L., Rusiñol, M., & Karatzas, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01275 · Cited by 136
Hashtag Recommendation for Multimodal Microblog Using Co-Attention Network
Zhang, Q., Wang, J., Huang, H., Huang, X., & Gong, Y. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/478 · Cited by 136
A Dataset for Multi-Target Stance Detection
Sobhani, P., Inkpen, D. Z., & Zhu, X. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2088 · Cited by 136
BDANN: BERT-Based Domain Adaptation Neural Network for Multi-Modal Fake News Detection
Zhang, T., Wang, D., Chen, H., Zeng, Z., Guo, W., Miao, C., & Cui, L. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206973 · Cited by 147
Cross-Modal Retrieval via Deep and Bidirectional Representation Learning
He, Y., Xiang, S., Kang, C., Wang, J., & Pan, C. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2558463 · Cited by 137
Dialog-based Interactive Image RetrievalPreprint
Guo, X., Wu, H., Cheng, Y., Rennie, S. J., Tesauro, G. J., & Feris, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.00145 · Cited by 135
Multitask Learning for Cross-Domain Image Captioning
Yang, M., Zhao, W., Xu, W. H., Feng, Y., Zhao, Z., Chen, X., & Lei, K. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2869276 · Cited by 134
Knowledge-aware Multimodal Dialogue Systems
Liao, L., Ma, Y., He, X., Hong, R., & Chua, T. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240605 · Cited by 134
Learn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136
Compressing Convolutional Neural Networks in the Frequency Domain
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., & Chen, Y. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939839 · Cited by 134
Retrospective Encoders for Video Summarization
Zhang, K., Grauman, K. L., & Sha, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_24 · Cited by 132
The N3XT Approach to Energy-Efficient Abundant-Data Computing
Aly, M. M. S., Wu, T. F., Bartolo, A., Malviya, Y. H., Hwang, W., Hills, G., Markov, I. L., Wootters, M. K., Shulaker, M. M., Wong, H. ‐. P., et al. (2018) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2882603 · Cited by 134
Reasoning Visual Dialogs With Structural and Partial Observations
Zheng, Z., Wang, W., Qi, S., & Zhu, S. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00683 · Cited by 131
Learning Deep Representations of Fine-grained Visual DescriptionsPreprint
Reed, S., Akata, Z., Schiele, B., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05395 · Cited by 156
Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text
Venugopalan, S., Hendricks, L. A., Mooney, R. J., & Saenko, K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1204 · Cited by 138
Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents
Zhang, R., Lee, H., & Radev, D. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1177 · Cited by 136
Creativity: Generating Diverse Questions Using Variational Autoencoders
Jain, U., Zhang, Z., & Schwing, A. G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.575 · Cited by 135
GLA: Global–Local Attention for Image Description
Li, L., Tang, S., Zhang, Y., Deng, L., & Tian, Q. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2751140 · Cited by 132
Pulling out all the tops with computer vision and deep learning
Macaluso, S., & Shih, D. (2018) · Journal of High Energy Physics · doi:10.1007/jhep10(2018)121 · Cited by 195
Feature Boosting Network For 3D Pose Estimation
Liu, J., Ding, H., Shahroudy, A., Duan, L., Jiang, X., Wang, G., & Kot, A. C. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894422 · Cited by 135
Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning
Wu, Q., Wang, P., Shen, C., Reid, I. D., & van den Hengel, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00639 · Cited by 134
Learning Dual Semantic Relations With Graph Attention for Image-Text Matching
Wen, K., Gu, X., & Cheng, Q. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3030656 · Cited by 131
Deconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 136
Reinforced Cross-modal Alignment for Radiology Report Generation
Qin, H., & Song, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.38 · Cited by 145
CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval
Wang, Z., Zhu, A., Xue, J., Wan, X., Liu, C., Wang, T., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548057 · Cited by 131
“R@1 R@5 R@10 CNN-RNN [19] 8.07 - 32.47 Neural Talk [25] 13.66 - 41.72 GNA-RNN [16] 19.05 - 53.64 IATV [15] 25.94 - 60.48 PWM-ATH [3] 27.14 49.45 61.02 Dual Path [37] 44.40 66.26 75.07 GLA [2] 43.58 66.93 76.26 CMPM-CMPC [33] 49.37 71.69 79.27 MIA [18] 53.10 75.00 82.90 A-GANet [17] 53.14 74.03…”
InfluentialImage Captioning with Deep Bidirectional LSTMs and Multi-Task Learning
Wang, C., Yang, H., & Meinel, C. (2018) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3115432 · Cited by 132
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132
“Among the generative models, maximum entropy language model (MELM) [31, 75, 76], machine translation (MT) sequenceto-sequence model, and gated recurrent neural networks (GRNN), derived from [77, 78] were constructed and evaluated.”
MethodMulti30K: Multilingual English-German Image Descriptions
Elliott, D., Frank, S., Sima’an, K., & Specia, L. (2016) · Workshop on Vision and Language (VL) · doi:10.18653/v1/w16-3210 · Cited by 129
Recursive Visual Attention in Visual Dialog
Niu, Y., Zhang, H., Zhang, M., Zhang, J., Lu, Z., & Wen, J.-R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00684 · Cited by 131
TableFormer: Table Structure Understanding with Transformers
Nassar, A., Livathinos, N., Lysak, M., & Staar, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00457 · Cited by 129
“The easiest network architectures are “image-encoder→ text-decoder” (IETD), similar to network architectures that try to provide captions to images [32].”
BackgroundNWPU-Captions Dataset and MLCA-Net for Remote Sensing Image Captioning
Cheng, Q., Huang, H., Xu, Y., Zhou, Y., Huan-ying, L., & Wang, Z. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3201474 · Cited by 129
Zhang, J., Du, J., & Dai, L.-R. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2844689 · Cited by 128
A long video caption generation algorithm for big video data retrieval
Ding, S., Qu, S., Xi, Y., & Wan, S. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.10.054 · Cited by 128
Modeling Deep Learning Accelerator Enabled GPUs
Raihan, A., Goli, N., & Aamodt, T. M. (2019) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass.2019.00016 · Cited by 128
Wang, Z., Zhu, A., Xue, J., Wan, X., Liu, C., Wang, T., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548166 · Cited by 128
“2 that the performance of the model with USEM is obviously better than the other substitutions, which indicate the effectiveness of USEM.”
MethodSemantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 139
“For example, the earlier attempts [10, 19, 40] present an end-to-end neural network system that encodes the visual content with CNN and decodes the sentence with RNN.”
CPTR: Full Transformer Network for Image CaptioningPreprint
Liu, W., Chen, S., Guo, L., Zhu, X., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10804 · Cited by 128
CapSal: Leveraging Captioning to Boost Semantics for Salient Object Detection
Zhang, L., Zhang, J., Lin, Z., Lu, H., & He, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00618 · Cited by 127
SemStyle: Learning to Generate Stylised Image Captions Using Unaligned Text
Mathews, A., Xie, L., & He, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00896 · Cited by 123
A multiscale residual pyramid attention network for medical image fusion
Fu, J., Li, W., Du, J., & Huang, Y. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.102488 · Cited by 127
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128
Large-Scale Datasets for Going Deeper in Image Understanding
Wu, J., He, Z., Zhao, B., Li, Y., Yan, B., Liang, R. S., Wang, W., Zhou, S., Lin, G., Fu, Y., et al. (2019) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00256 · Cited by 140
EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks
Zhang, P., Xue, J., Lan, C., Zeng, W., Gao, Z., & Zheng, N. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2937724 · Cited by 126
A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks
Xue, Y., Wang, Y., Liang, J., & Słowik, A. (2021) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2021.3084435 · Cited by 128
No More Pesky Learning Rate Guessing Games.Preprint
Smith, L. N. (2015) · arXiv (Cornell University) · Cited by 126
Image-Grounded Conversations: Multimodal Context for Natural Question and Response GenerationPreprint
Mostafazadeh, N., Brockett, C., Dolan, B., Galley, M., Gao, J., Spithourakis, G., & Vanderwende, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.08251 · Cited by 125
Deep Hierarchical Encoder–Decoder Network for Image Captioning
Xiao, X., Wang, L., Ding, K., Xiang, S., & Pan, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2915033 · Cited by 134
Distilling Knowledge Learned in BERT for Text Generation
Chen, Y.-C., Gan, Z., Yu, C., Liu, J., & Liu, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.705 · Cited by 151
Variations in Variational Autoencoders - A Comparative Evaluation
Wei, R., Garcia, C., El-Sayed, A., Peterson, V., & Mahmood, A. (2020) · IEEE Access · doi:10.1109/access.2020.3018151 · Cited by 125
Compression Artifacts Removal Using Convolutional Neural NetworksPreprint
Svoboda, P., Hradiš, M., Barina, D., & Zemčík, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.00366 · Cited by 136
Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
Sun, C., Shetty, S., Sukthankar, R., & Nevatia, R. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806226 · Cited by 124
Move Forward and Tell: A Progressive Generator of Video Descriptions
Xiong, Y., Dai, B., & Lin, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_29 · Cited by 120
Pix2seq: A Language Modeling Framework for Object DetectionPreprint
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 272
Answer-Type Prediction for Visual Question Answering
Kafle, K., & Kanan, C. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.538 · Cited by 123
Link the Head to the "Beak": Zero Shot Learning from Noisy Text Description at Part Precision
Elhoseiny, M., Zhu, Y., Zhang, H., & Elgammal, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.666 · Cited by 123
Chen, H., Zhang, H., Chen, P., Yi, J., & Hsieh, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1241 · Cited by 155
Recurrent Neural Networks Hardware Implementation on FPGAPreprint
Chang, A. X. M., Martini, B., & Culurciello, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05552 · Cited by 148
Han, Y., Wang, S., Ren, Y., Wang, C., Gao, P., & Chen, G. (2019) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi8060243 · Cited by 126
Chinese Image Captioning via Fuzzy Attention-based DenseNet-BiLSTM
Lu, H., Yang, R., Deng, Z., Zhang, Y., Gao, G., & Lan, R. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3422668 · Cited by 122
AI-assisted knowledge assessment techniques for adaptive learning environments
Minn, S. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100050 · Cited by 166
“…translation, language modeling, question answering (Chen et al., Chenet al.; Edunov et al., 2018; Dhingra et al., Salakhutdinov; Devlin, Chang, Lee, & Toutanova, 2018)), image classification, and computer vision (such as image captioning (Vinyals, Toshev, Bengio, & Erhan, 2015; Xu et al., 2015)).”
BackgroundPlappert, M., Mandery, C., & Asfour, T. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.006 · Cited by 126
Deep Learning for Image-to-Text Generation: A Technical Overview
He, X., & Deng, L. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2741510 · Cited by 126
The Long-Short Story of Movie Description
Rohrbach, A., Rohrbach, M., & Schiele, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24947-6_17 · Cited by 119
Metadata
- Authors
- Vinyals, O., Toshev, A., Bengio, S., & Erhan, D.
- Affiliations
- Google (United States) (US)
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3156-3164
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1895577753
- Cited by
- 6,318
- References
- 40
- Citations by year
- 2026: 228 · 2025: 464 · 2024: 547 · 2023: 598 · 2022: 589 · 2021: 747 · 2020: 748 · 2019: 770 · 2018: 656 · 2017: 471 · 2016: 329 · 2015: 158
- FWCI
- 220.64
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- image captioning, recurrent neural networks, generative models, machine translation, image description generation, PASCAL dataset, Flickr30k, COCO dataset, BLEU evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
