The DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index.
Show and tell: A neural image caption generator
Abstract
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They coupled a convolutional neural network for encoding of the image, with a recurrent neural network - specifically a long-short term memory net (LSTM) (Hochreiter and Schmidhuber (1997)) - for textual encoding.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Vinyals et al. (2015)
In the citing paper
“They used human-generated sentences as ground truth for training, and used the bilingual evaluation understudy (BLEU) metric for evaluation.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Vinyals et al. (2015)
In the citing paper
“Vinyals et al. (2015) provided textual explanation for images using an end-to-end image captioning framework.”
From Explainable artificial intelligence (XAI) in deep learning-based medical image analysis (2022) · cites this paper as Vinyals et al. (2015)
In the citing paper
“Similarly, an image and a natural language caption can capture complimentary but converging information about a scene [32, 35].”
From Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018) · cites this paper as [32]
In the citing paper
“Dependence on complex datasets including images and textual annotations”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [43]
In the citing paper
“Semantic relationship between images and textual descriptions is intuitive, even for nonexperts”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [43]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Self-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Flagged first, then by citations
3 of the 40 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- kuznetsova, 2014, Treetalk: Composition and compression of trees for image descriptions, ACL, 2
Cited by
Sorted by citations
Items 901–1,000 of 1,039
NN-Baton: DNN Workload Orchestration and Chiplet Granularity Exploration for Multichip Accelerators
Tan, Z., Cai, H., Dong, R., & Ma, K. (2021) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca52012.2021.00083 · Cited by 71
Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation
Yan, A., He, Z., Lü, X., Jiang, D., Chang, E., Gentili, A., McAuley, J., & Hsu, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.336 · Cited by 79
Sattarzadeh, A. R., Kutadinata, R. J., Pathirana, P. N., & Huynh, V. T. (2023) · Transportmetrica A Transport Science · doi:10.1080/23249935.2023.2236724 · Cited by 83
“Vinyals et al. (2015) presented the visual attention mechanism for caption generation and the integrating of CNN and RNN layers have also had impressive results in visual activity recognition (Donahue et al. 2015), sentiment analysis (Wang et al. 2016), video classification (Wu et al. 2015), and 3D…”
Fixed-point performance analysis of recurrent neural networks
Shin, S., Hwang, K., & Sung, W. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7471821 · Cited by 72
Rawat, Y. S., & Kankanhalli, M. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2984068 · Cited by 69
Task-Driven Dynamic Fusion: Reducing Ambiguity in Video Description
Zhang, X., Gao, K., Zhang, Y., Zhang, D., Li, J., & Tian, Q. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.662 · Cited by 68
Estimating Brain Connectivity With Varying-Length Time Lags Using a Recurrent Neural Network
Wang, Y., Lin, K., Qi, Y., Lian, Q., Feng, S., Wu, Z., & Pan, G. (2018) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2018.2842769 · Cited by 69
Shapeglot: Learning Language for Shape Differentiation
Achlioptas, P., Guibas, L. J., Goodman, N. D., Fan, J., & Hawkins, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00903 · Cited by 72
Compressing Visual-linguistic Model via Knowledge Distillation
Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., & Liu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00146 · Cited by 81
FMFN: Fine-Grained Multimodal Fusion Networks for Fake News Detection
Wang, J., Mao, H., & Li, H. (2022) · Applied Sciences · doi:10.3390/app12031093 · Cited by 76
“• NeuralTalk [39]: The model aims to produce captions for given images.”
BackgroundUniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Masry, A., Kavehzadeh, P., Long, X., Hoque, E., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.906 · Cited by 186
A Novel Convolutional Neural Network for Image Steganalysis With Shared Normalization
Wu, S., Zhong, S.-H., & Liu, Y. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2920605 · Cited by 72
Human Trajectory Prediction using Spatially aware Deep Attention ModelsPreprint
Varshneya, D., & Srinivasaraghavan, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09436 · Cited by 98
ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset
Rückert, J., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., Schäfer, H., Schmidt, C. S., Koitka, S., Pelka, O., Abacha, A. B., de Herrera, A. G. S., et al. (2024) · Scientific Data · doi:10.1038/s41597-024-03496-6 · Cited by 95
“Table 5 lists the ten most common UMLS concepts found in the dataset, excluding modality-related concepts that were set manually.”
Guiding Novice Web Workers in Making Image Descriptions Using Templates
Morash, V. S., Siu, Y.-T., Miele, J. A., Hasty, L., & Landau, S. (2015) · ACM Transactions on Accessible Computing · doi:10.1145/2764916 · Cited by 69
Liu, F., Ren, X., Liu, Y., Wang, H., & Sun, X. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1013 · Cited by 73
A Dataset for Telling the Stories of Social Media Videos
Gella, S., Lewis, M., & Rohrbach, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1117 · Cited by 69
Lyu, F., Wu, Q., Hu, F., Wu, Q., & Tan, M. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2894964 · Cited by 67
Deep learning-based method for vision-guided robotic grasping of unknown objects
Bergamini, L., Sposato, M., Pellicciari, M., Peruzzini, M., Calderara, S., & Schmidt, J. (2020) · Advanced Engineering Informatics · doi:10.1016/j.aei.2020.101052 · Cited by 70
A Simple Baseline for Audio-Visual Scene-Aware Dialog
Schwartz, I., Schwing, A. G., & Hazan, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01283 · Cited by 71
Eliminating Exposure Bias and Metric Mismatch in Multiple Object Tracking
Maksai, A., & Fua, P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00477 · Cited by 69
Img2Mol – accurate SMILES recognition from molecular graphical depictions
Clevert, D.-A., Le, T., Winter, R., & Montanari, F. (2021) · Chemical Science · doi:10.1039/d1sc01839f · Cited by 74
In Defense of Scene Graphs for Image Captioning
Nguyen, K., Tripathi, S., Du, B., Guha, T., & Nguyen, T. Q. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00144
What is the best RNN-cell structure to forecast each time series behavior?
Khaldi, R., Afia, A. E., Chiheb, R., & Tabik, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119140 · Cited by 79
Generating Image Captions Using Bahdanau Attention Mechanism and Transfer Learning
Ayoub, S., Gulzar, Y., Reegu, F. A., & Turaev, S. (2022) · Symmetry · doi:10.3390/sym14122681 · Cited by 69
Deep Learning for Algorithm Portfolios
Loreggia, A., Malitsky, Y., Samulowitz, H., & Saraswat, V. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10170 · Cited by 69
Li, L., Jing, H., Tong, H., Yang, J., He, Q., & Chen, B.-C. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054200 · Cited by 76
DeltaDelta neural networks for lead optimization of small molecule potency
Jiménez-Luna, J., Pérez‐Benito, L., Martínez-Rosell, G., Sciabola, S., Torella, R., Tresadern, G., & Fabritiis, G. D. (2019) · Chemical Science · doi:10.1039/c9sc04606b · Cited by 73
CNN+CNN: Convolutional Decoders for Image CaptioningPreprint
Wang, Q., & Chan, A. B. (2018) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1805.09019 · Cited by 95
Interest Point Detector and Feature Descriptor Survey
Krig, S. (2014) · Apress eBooks · doi:10.1007/978-1-4302-5930-5_6 · Cited by 74
A Systematic Literature Review on Image Captioning
Staniūtė, R., & Šešok, D. (2019) · Applied Sciences · doi:10.3390/app9102024 · Cited by 68
Synchronous Transformers for end-to-end Speech Recognition
Tian, Z., Yi, J., Bai, Y., Tao, J., Zhang, S., & Wen, Z. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054260 · Cited by 68
Length-Controllable Image Captioning
Deng, C., Ding, N., Tan, M., & Wu, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_42 · Cited by 70
Fake news detection for epidemic emergencies via deep correlations between text and images
Zeng, J., Zhang⋆, Y., & Ma, X. (2020) · Sustainable Cities and Society · doi:10.1016/j.scs.2020.102652 · Cited by 70
Boosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph
Zhao, W., & Wu, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301279 · Cited by 73
A Novel SVM-Based Decoder for Remote Sensing Image Captioning
Hoxha, G., & Melgani, F. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3105004 · Cited by 77
Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 86
“In other words, the model learns the implicit information from the environment, estimates the navigation, and predicts the next position.”
BackgroundFootball Action Recognition Using Hierarchical LSTM
Tsunoda, T., Komori, Y., Matsugu, M., & Harada, T. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.25 · Cited by 70
A Hierarchical Grocery Store Image Dataset With Visual and Semantic Labels
Klasson, M., Zhang, C., & Kjellström, H. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00058 · Cited by 73
Integrating Long Short-Term Memory and Genetic Algorithm for Short-Term Load Forecasting
Santra, A. S., & Lin, J.-L. (2019) · Energies · doi:10.3390/en12112040 · Cited by 69
Multilingual Image Description with Neural Sequence ModelsPreprint
Elliott, D., Frank, S., & Hasler, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.04709 · Cited by 85
Learning Language-Visual Embedding for Movie Understanding with Natural-LanguagePreprint
Torabi, A., Tandon, N., & Sigal, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08124 · Cited by 68
FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Rotstein, N., Bensaïd, D., Brody, S., Ganz, R., & Kimmel, R. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00559 · Cited by 66
“Subsequently, the encoder-decoder framework, which essentially translates an image into a sentence, became one of the favored approaches [20,66].”
BackgroundFFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer
Chen, X., Zhang, W., Xu, P., Zhao, Z., Zheng, Y., Shi, D., & He, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01101-z · Cited by 72
Representation and Computation in Cognitive Models
Forbus, K. D., Liang, C., & Rabkina, I. (2017) · Topics in Cognitive Science · doi:10.1111/tops.12277 · Cited by 67
Video Captioning with Guidance of Multimodal Latent Topics
Chen, S., Chen, J., Jin, Q., & Hauptmann, A. G. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123420 · Cited by 69
Personalized Review Generation By Expanding Phrases and Attending on Aspect-Aware Representations
Ni, J., & McAuley, J. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2112 · Cited by 71
Entity-aware Image Caption Generation
Lu, D., Whitehead, S., Huang, L., Ji, H., & Chang, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1435 · Cited by 75
Yang, M., Liu, J., Shen, Y., Zhao, Z., Chen, X., Wu, Q., & Li, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3028651 · Cited by 67
Learning to Generate Clinically Coherent Chest X-Ray Reports
Lovelace, J., & Mortazavi, B. J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.110 · Cited by 66
Dual self-attention with co-attention networks for visual question answering
Liu, Y., Zhang, X., Zhang, Q., Li, C., Huang, F., Tang, X., & Li, Z. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107956 · Cited by 68
Text to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 78
Bhooshan, R. S., Kumaraswamy, S., & Sasidharan, S. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00693-9 · Cited by 72
“Later advanced image captioning systems are designed using deep neural architectures that uses a Convolutional Neural Network (CNN) as encoder for visual feature extraction and a Recurrent Neural Network (RNN) as decoder for text generation [3].”
MethodReale-Nosei, G., Amador-Domínguez, E., & Serrano, E. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103264 · Cited by 72
Fine-Grained Video Captioning for Sports Narrative
Yu, H., Cheng, S., Ni, B., Wang, M., Zhang, J., & Yang, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00629 · Cited by 67
Diverse and Coherent Paragraph Generation from Images
Chatterjee, M., & Schwing, A. G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_45 · Cited by 67
Image annotation: Then and now
Bhagat, P. K., & Choudhary, P. (2018) · Image and Vision Computing · doi:10.1016/j.imavis.2018.09.017 · Cited by 66
Hierarchically-Attentive RNN for Album Summarization and Storytelling
Yu, L., Bansal, M., & Berg, T. L. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1101 · Cited by 67
Guiding Long-Short Term Memory for Image Caption GenerationPreprint
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.04942 · Cited by 67
Bringing Alive Blurred Moments
Purohit, K., Shah, A., & Rajagopalan, A. N. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00699 · Cited by 64
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
Harwath, D. F., Recasens, A., Surís, D., Chuang, G., Torralba, A. B., & Glass, J. R. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01205-0 · Cited by 79
Adversarial Attribute-Text Embedding for Person Search With Natural Language Query
Zha, Z.-J., Liu, J., Chen, D., & Wu, F. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972168 · Cited by 63
A New Image Captioning Approach for Visually Impaired People
Makav, B., & Kılıç, V. (2019) · National Conference on Electrical and Electronics Engineering (ELECO) · doi:10.23919/eleco47770.2019.8990630 · Cited by 69
Auto-encoding and Distilling Scene Graphs for Image Captioning
Yang, X., Zhang, H., & Cai, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3042192 · Cited by 68
Huang, B., Ji, Z., Zhai, R., Xiao, C., Yang, F., Yang, B., & Wang, Y. (2021) · GPS Solutions · doi:10.1007/s10291-021-01115-0 · Cited by 70
Fine-grained Image Captioning with CLIP Reward
Cho, J., Yoon, S., Kale, A., Dernoncourt, F., Bui, T., & Bansal, M. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.39 · Cited by 96
Returnn: The RWTH extensible training framework for universal recurrent neural networks
Doetsch, P., Zeyer, A., Voigtlaender, P., Kulikov, I., Schlüter, R., & Ney, H. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953177 · Cited by 65
Towards Robust Deep Neural Networks with BANG
Rozsa, A., Günther, M., & Boult, T. E. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00093 · Cited by 69
A Neural Approach to Pun Generation
Yu, Z., Tan, J., & Wan, X. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1153 · Cited by 66
DNPU: An Energy-Efficient Deep-Learning Processor with Heterogeneous Multi-Core Architecture
Shin, D., Lee, J., Lee, J., Lee, J., & Yoo, H. (2018) · IEEE Micro · doi:10.1109/mm.2018.053631145 · Cited by 65
Improving Image Captioning by Leveraging Knowledge Graphs
Zhou, Y., Sun, Y., & Honavar, V. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00036 · Cited by 65
pix2code: Generating Code from a Graphical User Interface ScreenshotPreprint
Beltramelli, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07962 · Cited by 110
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech
Kamper, H., Settle, S., Shakhnarovich, G., & Livescu, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-502 · Cited by 64
Neural Wikipedian: Generating Textual Summaries from Knowledge Base Triples
Vougiouklis, P., Elsahar, H., Kaffee, L.-A., Gravier, C., Laforest, F., Hare, J. S., & Simperl, E. P. B. (2018) · Journal of Web Semantics · doi:10.1016/j.websem.2018.07.002 · Cited by 66
Decoupled Novel Object Captioner
Wu, Y., Zhu, L., Jiang, L., & Yang, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240640 · Cited by 62
Automatic Arabic Image Captioning using RNN-LSTM-Based Language Model and CNN
Almuzaini, H. A., Tasniem, N., & Benhidour, H. (2018) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2018.090610 · Cited by 69
A Multi-Level Attention Model for Remote Sensing Image Captions
Li, Y., Fang, S., Jiao, L., Liu, R., & Shang, R. (2020) · Remote Sensing · doi:10.3390/rs12060939 · Cited by 72
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
Huang, J.-H., Yang, C.-H. H., Liu, F., Tian, M., Liu, Y.-C., Wu, T.-W., Lin, I.-H., Wang, K., Morikawa, H., Chang, H.-H., et al. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00249 · Cited by 65
Double-Stream Position Learning Transformer Network for Image Captioning
Jiang, W., Zhou, W., & Hu, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3181490 · Cited by 66
“The compared models include NIC [4], SCST [30], LSTM-A [51], RFNet [12], Up-Down [6], HAN [13], GCN-LSTM [10], SGAE [35], NADGCN [14], ETA [11], AoANet [7], Trans-former [52], MT sv [40], M 2 Transformer [8], X-Transformer [39], GET [42] and APN [38].”
BackgroundMethodHistopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis
Abdullakutty, F. C., Akbari, Y., Al-Maadeed, S. A., Bouridane, A., Talaat, I. M., & Hamoudi, R. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1450103 · Cited by 65
“Image captioning in XAI involves generating descriptive text for images, which aids in elucidating the decision-making process of AI models (120).”
MethodDeep Video Stabilization Using Adversarial Networks
Xu, S., Hu, J., Wang, M., Mu, T., & Hu, S. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13566 · Cited by 65
Automatic Sleep Staging Employing Convolutional Neural Networks and Cortical Connectivity Images
Chriskos, P., Frantzidιs, C. A., Gkivogkli, P. T., Bamidis, P. D., & Kourtidou‐Papadeli, C. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2899781 · Cited by 66
Cui, W. H., Wang, F., He, X. J., Zhang, D., Xu, X., Meng, Y., Wang, Z., & Huang, J. (2019) · Remote Sensing · doi:10.3390/rs11091044 · Cited by 67
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
Li, Q., Fu, J., Yu, D., Mei, T., & Luo, J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1164 · Cited by 71
Huang, X., Yan, F., Xu, W., & Li, M. (2019) · IEEE Access · doi:10.1109/access.2019.2947134 · Cited by 65
Reference Based LSTM for Image Captioning
Chen, M., Ding, G., Zhao, S., Chen, H., Liu, Q., & Han, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11198 · Cited by 63
AnchorNet: A Weakly Supervised Network to Learn Geometry-Sensitive Features for Semantic Matching
Novotný, D., Larlus, D., & Vedaldi, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.306 · Cited by 65
Joint Learning of Object and Action Detectors
Kalogeiton, V. S., Weinzaepfel, P., Ferrari, V., & Schmid, C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.219 · Cited by 61
Cross-Domain Image Captioning via Cross-Modal Retrieval and Model Adaptation
Zhao, W., Wu, X., & Luo, J. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3042086 · Cited by 67
Automatic Image Annotation Based on Deep Learning Models: A Systematic Review and Future Challenges
Adnan, M. M., Rahim, M. S. M., Rehman, A. U., Mehmood, Z., Saba, T., & Naqvi, R. A. (2021) · IEEE Access · doi:10.1109/access.2021.3068897 · Cited by 67
Deep Learning Approaches Based on Transformer Architectures for Image Captioning Tasks
Castro, R., Pineda, I., Lim, W., & Morocho-Cayamcela, M. E. (2022) · IEEE Access · doi:10.1109/access.2022.3161428 · Cited by 69
Underwater image captioning: Challenges, models, and datasets
Li, H., Wang, H., Zhang, Y., Li, L., & Ren, P. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.12.002 · Cited by 66
Low latency RNN inference with cellular batching
Gao, P., Yu, L., Wu, Y., & Li, J. (2018) · Proceedings of the Thirteenth EuroSys Conference · doi:10.1145/3190508.3190541 · Cited by 66
Recurrent Neural Networks for Semantic Instance Segmentation
Salvador, A., Bellver, M., Campos, V., Baradad, M., Marques, F., Torres, J., & Giro-i-Nieto, X. (2017) · arXiv · doi:10.48550/arxiv.1712.00617 · Cited by 62
Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
Sachdeva, E., Agarwal, N., Chundi, S., Roelofs, S., Li, J., Kochenderfer, M., Choi, C., & Dariush, B. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00734 · Cited by 63
“For the captioning baselines, Scan2cap [6] outperforms S&T [37] because Scan2Cap uses 3D point clouds and 3D bounding box features, which are more informative than 2D features.”
MethodA multimodal fusion approach for image captioning
Zhao, D., Chang, Z., & Guo, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.004 · Cited by 61
Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning
Dong, X., Long, C., Xu, W., & Xiao, C. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475439 · Cited by 74
Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation
Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., & Chang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02000 · Cited by 60
GenAssist: Making Image Generation Accessible
Huh, M., Peng, Y.-H., & Pavel, A. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606735 · Cited by 67
“Early work hired crowd workers to create alt text [6, 72], while recent research has developed machine-learning-based systems that automatically generate image descriptions [34, 71, 81].”
BackgroundErdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
“Figure 3: Illustration of the Neural Image Captioning (NIC) image-to-text description generation, after (Vinyals et al., 2015).”
BackgroundMethodInfluential
Metadata
- Authors
- Vinyals, O., Toshev, A., Bengio, S., & Erhan, D.
- Affiliations
- Google (United States) (US)
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3156-3164
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1895577753
- Cited by
- 6,318
- References
- 40
- Citations by year
- 2026: 228 · 2025: 464 · 2024: 547 · 2023: 598 · 2022: 589 · 2021: 747 · 2020: 748 · 2019: 770 · 2018: 656 · 2017: 471 · 2016: 329 · 2015: 158
- FWCI
- 220.64
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- image captioning, recurrent neural networks, generative models, machine translation, image description generation, PASCAL dataset, Flickr30k, COCO dataset, BLEU evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
