A Comprehensive Survey of Deep Learning for Image Captioning

Hossain, MD. Zakir; Sohel, Ferdous; Shiratuddin, Mohd Fairuz; Laga, Hamid

doi:10.1145/3295748

Cited by 879Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesGenerative Adversarial Networks and Image Synthesis

A Comprehensive Survey of Deep Learning for Image Captioning

Hossain, MD. Zakir, Sohel, Ferdous, Shiratuddin, Mohd Fairuz, Laga, Hamid·2019·ACM Computing Surveys

DOI 10.1145/3295748

Open this paper in CoChat — free account

Abstract

Generating a description of an image is called image captioning. Image captioning requires recognizing the important objects, their attributes, and their relationships in an image. It also needs to generate syntactically and semantically correct sentences. Deep-learning-based techniques are capable of handling the complexities and challenges of image captioning. In this survey article, we aim to present a comprehensive review of existing deep-learning-based image captioning techniques. We discuss the foundation of the techniques to analyze their performances, strengths, and limitations. We also discuss the datasets and the evaluation metrics popularly used in deep-learning-based automatic image captioning.

Integrity signals

Registry

Crossref · 10.1145/3295748

Cites

166 of 166 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Most prior works have relied on metrics inspired by natural language generation, treating radiology report text as generic text.
“Most prior works have relied on a set of metrics inspired by similar setups in natural language generation, where radiology report text is treated as generic text.22”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [22]
Many works are using cross-modal deep generative models such as GAN in data synthesis, such as from video to audio, from text to image and vice versa.
“If we take a broader view, many works are using cross-modal deep generative models (such as GAN) in data synthesis, such as from video to audio [267], from text to image and vice versa [268,269].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [268]

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2× · see citing works
“The architecture of this method has ResNet [53] and two LSTMs called Skel-LSTM and Attr-LSTM.”
Method
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5× · see citing works
“Kiros et al. [69] proposed a method that is built on a Log-Bilinear model and used AlexNet to extract visual features.”
BackgroundMethodInfluential
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3× · see citing works
“LSTM [54] networks are a type of RNN that have special units in addition to standard units.”
BackgroundMethod
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · cited 2× · see citing works
“Moreover, recent object detection and classification methods [73, 127] show that deep, hierarchical methods are better at learning than shallower ones.”
BackgroundMethod
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1× · see citing works
“For example, Convolutional Neural Networks (CNN) [79] are widely used for feature learning, and a classifier such as Softmax is used for classification.”
Method
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · cited 4× · see citing works
“The bottom up attention mechanism uses Faster R-CNN [116] for region proposals that can select salient regions of an image .”
MethodInfluential
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 1× · see citing works
“d techniques and (2) Deep machine learning based techniques. In traditional machine learning, hand crafted features such as Local Binary Patterns (LBP) [107], Scale-Invariant Feature Transform (SIFT) [87], the Histogram of Oriented Gradients (HOG) [27], and a combination of such features are widely used. In these tech…”
Background
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · cited 1× · see citing works
“In traditional machine learning, hand crafted features such as Local Binary Patterns (LBP) [107], Scale-Invariant Feature Transform (SIFT) [87], the Histogram of Oriented Gradients (HOG) [27], and a combination of such features are widely used.”
Background
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Latent dirichlet allocation
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Fast R-CNN
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1× · see citing works
“between a pair of networks: the Generator and the Discriminator. GANs have already been used successfully in a variety of applications, including image captioning[26, 126], image to image translation [56], text to image synthesis [15, 115], and text generation [36,145]. There are two issues with GAN. First, GAN can wo…”
Background
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2× · see citing works
“and test data consists of 2,000 and 1,000 images respectively. Each image contains romantic, humorous, and factual captions. 4.2 Evaluation Metrics 4.2.1 BLEU. BLEU (Bilingual evaluation understudy) [110] is a metric that is used to measure the quality of machine generated text. Individual text segments are compared w…”
MethodInfluential
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1× · see citing works
“In addition to this, in this method, each filter of a convolutional layer acts as semantic detectors [159] while other methods use external sources for obtaining semantic information.”
Method
Latent Dirichlet Allocation
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992 · cited 1× · see citing works
“They are then passed to a classifier such as Support Vector Machines (SVM) [17] in order to classify an object.”
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 1× · see citing works
“Another network, Gated Recurrent Unit (GRU) [25] has a similar structure to LSTM but it does not use separate memory cells and uses fewer gates to control the flow of information.”
Background
Untitled work
(2004) · Cited by 9,718 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 6× · see citing works
“The top-down paradigms [23, 31, 65, 93, 94, 142, 152] are used for extracting visual features first and then convert them into words.”
BackgroundMethodInfluential
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · cited 1× · see citing works
“Therefore, LSTM is still facing challenges in generating long length sentences [7, 24].”
Background
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 5× · see citing works
“[23] VGGNet RNN VS,SL,WS,EDA Fang et al. 2015 [33] AlexNet, VGGNet MELM VS,SL,WS,CA Jia et al. 2015 [59] VGGNet LSTM VS,SL,WS,EDA Karpathy et al. 2015 [65] VGGNet RNN MS,SL,WS,EDA Vinyals et al. 2015 [142] GoogLeNet LSTM VS,SL,WS,EDA Xu et al. 2015 [152] AlexNet LSTM VS,SL,WS,EDA,AB Jin et al. 2015 [61] VGGNet LSTM VS…”
BackgroundMethodInfluential
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1× · see citing works
“sequence to sequence learning. Several neural language models such as neural probabilistic language model [11], log-bilinear models [105], skip-gram models [98], and recurrent neural networks (RNNs) [99] have been proposed for learning sequence to sequence tasks. RNNs have widely been used in various sequence learning…”
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 1× · see citing works
“n evaluation metrics. You et al. [156] proposed a semantic attention-based image captioning method. The method providesadetailed,coherentdescriptionofsemanticallyimportantobjects.Thetop-downparadigms [23, 31, 65, 93, 94, 142, 152] are used for extracting visual features first and then convert them into words. In botto…”
Method
Untitled work
(1999) · Cited by 5,147 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 2× · see citing works
“Visual Genome dataset [72] is another dataset for image captioning.”
Method
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852 · cited 1× · see citing works
“generating image captions even for open domain images. It can detect a diverse set of visual concepts and generate captions for celebrities and landmarks. It uses an external knowledge base Freebase [16] in recognizingabroadrangeofentitiessuchascelebritiesandlandmarks. Aseriesofhumanjudgments are applied for evaluatin…”
Method
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2× · see citing works
“ertain key entities such as celebrities and landmarks that are out of their scope. The generated captions of these methods are evaluated on automatic metrics such as BLEU [110], METEOR [1], and CIDEr [139]. These evaluation metrics have already shown good results on these methods. However, in terms of performance ther…”
BackgroundMethodInfluential
Untitled work
(2005) · Cited by 4,152 · see citing works
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · cited 1× · see citing works
“method can generate multiple captions for a single image and showed impressive improvements in generating diverse captions. GANs have limitations in backpropagating the discrete data. Gumbel sampler [58, 91] is used to overcome the discrete data problem. The two main parts of this adversarial network are the generator…”
Method
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451 · see citing works
A maximum entropy approach to natural language processing
Berger, A. L., Pietra, V. J. D., & Pietra, S. A. D. (1996) · doi:10.5555/234285.234289 · Cited by 3,083 · see citing works
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · Neural Information Processing Systems · doi:10.48550/arxiv.1606.05328 · Cited by 3,066 · cited 1× · see citing works
“ucture of the sentences and they are faster in processing than LSTMs. Therefore, recently, convolutional architectures are used in other sequence to sequence tasks, e.g., conditional image generation [137] and machine translation [42,43,138]. Inspired by the above success of CNNs in sequence learning tasks, Gu et al.…”
Method
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 1× · see citing works
“Therefore, recently, convolutional architectures are used in other sequence to sequence tasks, e.g., conditional image generation [137] and machine translation [42, 43, 138].”
Method
Minimum error rate training in statistical machine translation
Och, F. J. (2003) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075096.1075117 · Cited by 2,728 · cited 1× · see citing works
“Minimum Error rate training (MERT) [106] is used to learn these weights.”
Method
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,646 · see citing works
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · cited 1× · see citing works
“e Generator and the Discriminator. GANs have already been used successfully in a variety of applications, including image captioning[26, 126], image to image translation [56], text to image synthesis [15, 115], and text generation [36,145]. There are two issues with GAN. First, GAN can work well in generating natural…”
Background
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,342 · see citing works
Generating Typed Dependency Parses from Phrase Structure Parses
de Marneffe, M.-C., MacCartney, B., & Manning, C. D. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3wtkaidk6ea8 · Cited by 2,289 · cited 1× · see citing works
“thod works at a finer level and embeds fragments of images and fragments of sentences. This method breaks down the images into a number of objects and sentences into a dependency tree relations (DTR) [28] and reasons about their latent, inter-modal alignment. It shows that the method achieves significant improvements…”
Method
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · cited 1× · see citing works
“0.580 0.439 0.332 0.266 - 1.085 - Gan et al. 2017 [41] VS,SL,WS,CA,SCB 0.741 0.578 0.444 0.341 0.261 - 1.041 - Zhang et al. 2017 [161] VS,ODL,WS,EDA - - - 0.344 0.267 0.558 1.162 - Rennie et al. 2017 [120] VS,ODL,WS,EDA - - - .319 0.255 0.543 1.06 - Yao et al. 2017 [155] VS,SL,WS,EDA,SCB 0.734 0.567 0.430 0.326 0.254…”
Influential
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.667 · Cited by 2,073 · cited 2× · see citing works
“[21, 59, 61, 144, 150, 152] have performed experiments using the dataset.”
Method
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 1× · see citing works
“[3] is a new caption evaluation metric based on semantic concept.”
Background
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,859 · cited 4× · see citing works
“A general approach of this category is to analyze the visual content of the image first and then generate image captions from the visual content using a language model [70, 152, 155, 156].”
BackgroundMethodInfluential
The Concrete Distribution: A Continuous Relaxation of Discrete Random VariablesPreprint
Maddison, C. J., Mnih, A., & Teh, Y. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00712 · Cited by 1,771 · cited 1× · see citing works
“Gumbel sampler [58, 91] is used to overcome the discrete data problem.”
Method
Actor-critic algorithms
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 1,764 · see citing works
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · cited 1× · see citing works
“Lu et al. [88] proposed another attention-based image captioning method.”
Method
International Conference on Learning Representations (ICLR)
Abbas, M., Kivinen, J., & Raiko, T. (2016) · Cited by 1,632 · see citing works
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · cited 1× · see citing works
“next word is predicted given the image and all the previously generated ground-truth words. Therefore, the generated captions look-like ground-truth captions. This phenomenon is called exposure bias [10] problem. (ii) Evaluation metrics at test time are non-differentiable. Ideally sequence models for image captioning…”
Background
Understanding inverse document frequency: on theoretical arguments for IDF
Robertson, S. E. (2004) · Journal of Documentation · doi:10.1108/00220410410560582 · Cited by 1,550 · cited 1× · see citing works
“However, CIDEr achieves human consensus using term frequency-inverse document frequency (TF-IDF) [121].”
Background
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
DRAW: A Recurrent Neural Network For Image GenerationPreprint
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 1,394 · cited 1× · see citing works
“However, Johnson et al. [62] use a differential, spatial soft attention mechanism [49, 57] and bilinear interpolation [57] instead of ROI pooling mechanism [44].”
Background
Unifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsPreprint
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,386 · cited 2× · see citing works
“ceptance Date: October 2018. A Comprehensive Survey of Deep Learning for Image Captioning 0:23 Reference Datasets Evaluation Metrics Kiros et al. 2014 [69] IAPR TC-12,SBU BLEU, PPLX Kiros et al. 2014 [70] Flickr 8K, Flickr 30K R@K, mrank Mao et al. 2014 [95] IAPR TC-12, Flickr 8K/30K BLEU, R@K, mrank Karpathy et al. 2…”
BackgroundMethod
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · cited 4× · see citing works
“, No. 0, Article 0. Acceptance Date: October 2018. 0:24 Hossain et al. Fig. 13. Captions generated by Jia et al. [59] on some sample images from the Flickr8k dataset. 4.1.3 Flickr8K Dataset. Flickr8k [55] is a popular dataset and has 8000 images collected from Flickr. The training data consists of 6000 images, the tes…”
BackgroundMethodInfluential
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
(2013) · doi:10.18653/v1/d13-1 · Cited by 1,296 · see citing works
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,240 · cited 2× · see citing works
“of blank slots to generate captions. In these approaches, different objects, attributes, actions are detected first and then the blank spaces in the templates are filled. For example, Farhadi et al. [34] use a triplet of scene elements to fill the template slots for generating image captions. Li et al. [80] extract th…”
BackgroundMethod
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · cited 1× · see citing works
“Mao et al. [92] proposed a special type of text generation method for images.”
Method
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · cited 1× · see citing works
“The neural language models have limitations to handle a large amount of data and are inefficient to work with long term memory [64].”
Background
A Framework for Multiple-Instance Learning
Maron, O., & Lozano‐Pérez, T. (1997) · Cited by 1,175 · see citing works
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.494 · Cited by 1,161 · cited 3× · see citing works
“However, Johnson et al. [62] use a differential, spatial soft attention mechanism [49, 57] and bilinear interpolation [57] instead of ROI pooling mechanism [44].”
BackgroundMethod
Image retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101 · cited 1× · see citing works
“PICE. SPICE (Semantic Propositional Image Caption Evaluation) [3] is a new caption evaluation metric based on semantic concept. It is based on a graph-based semantic representation called scene-graph [63, 123]. This graph can extract the information of different objects, attributes and their relationships from the ima…”
Method
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · cited 2× · see citing works
“This method can generate a description for an specific object or region that is called referring expression [37, 46, 68, 102, 103, 136, 141].”
Method
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)Preprint
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 1,039 · cited 3× · see citing works
“Mao et al. [94] proposed a multimodal Recurrent Neural Network (m-RNN) method for generating novel image captions.”
BackgroundMethod
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 879 · see citing works
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2587640 · Cited by 875 · cited 1× · see citing works
“Generating well-formed sentences requires both syntactic and semantic understanding of the language [143].”
Background
Deep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 802 · cited 1× · see citing works
“n Metrics Kiros et al. 2014 [69] IAPR TC-12,SBU BLEU, PPLX Kiros et al. 2014 [70] Flickr 8K, Flickr 30K R@K, mrank Mao et al. 2014 [95] IAPR TC-12, Flickr 8K/30K BLEU, R@K, mrank Karpathy et al. 2014 [66] PASCAL1K, Flickr 8K/30K R@K, mrank Mao et al. 2015 [94] IAPR TC-12, Flickr 8K/30K, MS COCO BLEU, R@K, mrank Chen e…”
Background
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · cited 2× · see citing works
“Most sentence-image multimodal methods [38, 66, 70, 128] use pre-computed word embedding vectors to initialize their model.”
Method
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 752 · cited 2× · see citing works
“A general approach of this category is to analyze the visual content of the image first and then generate image captions from the visual content using a language model [70, 152, 155, 156].”
BackgroundMethod
Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737 · cited 1× · see citing works
“There are some cases where an increase in BLEU score does not mean that the quality of the generated text is good [82].”
Background
SALICON: Saliency in Context
Jiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729 · cited 1× · see citing works
“For experiments, it uses SALICON dataset [60] and achieves good results.”
Method
Eye Movement Analysis for Activity Recognition Using Electrooculography
Bulling, A., Ward, J. A., Gellersen, H. W., & Tröster, G. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.86 · Cited by 720 · see citing works
Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization
Murray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2005) · Meeting of the Association for Computational Linguistics · Cited by 716 · see citing works
Learning photographic global tonal adjustment with a database of input/output image pairs
Bychkovsky, V., Paris, S., Chan, E., & Durand, F. (2011) · doi:10.1109/cvpr.2011.5995413 · Cited by 666 · see citing works
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
Multi-task Sequence to Sequence LearningPreprint
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06114 · Cited by 606 · cited 1× · see citing works
“It uses multitask sequence to sequence training [89] for identifying the style factors and then add these factors at run time for generating attractive captions.”
Method
Gray Scale and Rotation Invariant Texture Classification with Local Binary Patterns
Ojala, T., Pietikäinen, M., & Mäenpää, T. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45054-8_27 · Cited by 600 · cited 1× · see citing works
“ategories: (1) Traditional machine learning based techniques and (2) Deep machine learning based techniques. In traditional machine learning, hand crafted features such as Local Binary Patterns (LBP) [107], Scale-Invariant Feature Transform (SIFT) [87], the Histogram of Oriented Gradients (HOG) [27], and a combination…”
Method
Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. (2014) · Cited by 593 · see citing works
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016) · International Conference on Learning Representations · Cited by 572 · see citing works
Three new graphical models for statistical language modelling
Mnih, A., & Hinton, G. E. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273577 · Cited by 569 · cited 1× · see citing works
“Several neural language models such as neural probabilistic language model [11], log-bilinear models [105], skip-gram models [98], and recurrent neural networks (RNNs) [99] have been proposed for learning sequence to sequence tasks.”
Learning photographic global tonal adjustment with a database of input / output image pairs
Bychkovsky, V., Paris, S., Chan, E., & Durand, F. (2011) · doi:10.1109/cvpr.2011.5995332 · Cited by 531 · see citing works
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G. A., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011) · doi:10.1109/cvpr.2011.5995466 · Cited by 521 · see citing works
Mind's eye: A recurrent visual representation for image caption generation
Chen, X., & Zitnick, C. L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298856 · Cited by 496 · cited 3× · see citing works
“The top-down paradigms [23, 31, 65, 93, 94, 142, 152] are used for extracting visual features first and then convert them into words.”
BackgroundMethod
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Dai, B., Fidler, S., Urtasun, R., & Lin, D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.323 · Cited by 483 · cited 3× · see citing works
“Dai et al. [26] also proposed a GAN based image captioning method.”
Method
A Convolutional Encoder Model for Neural Machine Translation
Gehring, J., Auli, M., Grangier, D., & Dauphin, Y. (2016) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p17-1012 · Cited by 470 · cited 1× · see citing works
“Therefore, recently, convolutional architectures are used in other sequence to sequence tasks, e.g., conditional image generation [137] and machine translation [42, 43, 138].”
Method
Guiding the Long-Short Term Memory Model for Image Caption Generation
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.277 · Cited by 469 · cited 3× · see citing works
“[21, 59, 61, 144, 150, 152] have performed experiments using the dataset.”
Background
Semantic Compositional Networks for Visual Captioning
Gan, Z., Gan, C., He, X., Pu, Y., Tran, K., Gao, J., Carin, L., & Deng, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.127 · Cited by 462 · see citing works
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · cited 1× · see citing works
“Liu et al. [85] introduced a new captions evaluation metric that is a good choice by human raters.”
Background
Convolutional Image Captioning
Aneja, J., Deshpande, A., & Schwing, A. G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00583 · Cited by 426 · see citing works
Explain Images with Multimodal Recurrent Neural NetworksPreprint
Mao, J., Xu, W., Yang, Y., Wang, J., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.1090 · Cited by 423 · cited 2× · see citing works
“Many image captioning methods [66, 69, 95] are built on recurrent neural networks at the contemporary times.”
Background
Midge: Generating Image Descriptions From Computer Vision Detections
Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T. L., & Daumé, H. (2012) · Cited by 421 · see citing works
Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
Wu, Q., Shen, C., Wang, P., Dick, A., & van den Hengel, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2708709 · Cited by 408 · cited 5× · see citing works
“Image captioning methods such as [22, 65, 142, 144, 150] use this dataset for their experiments.”
BackgroundMethodInfluential
Learning to Recognize Daily Actions Using Gaze
Fathi, A., Li, Y., & Rehg, J. M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_23 · Cited by 396 · cited 1× · see citing works
“Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160].”
Background
Composing Simple Image Descriptions using Web-scale N-grams
Li, S., Kulkarni, G. A., Berg, T. L., Berg, A. C., & Choi, Y. (2011) · Cited by 364 · see citing works
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, B., Çakıcı, R., Elliott, D., Erdem, A., Erdem, E., İkizler-Cinbiş, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 350 · cited 1× · see citing works
“So far, only three survey papers [8, 13, 75] have been published on this research topic.”
Background
MaskGAN: Better Text Generation via Filling in the______Preprint
Fedus, W., Goodfellow, I., & Dai, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07736 · Cited by 345 · cited 1× · see citing works
“GANs have already been used successfully in a variety of applications, including image captioning[26, 126], image to image translation [56], text to image synthesis [15, 115], and text generation [36, 145].”
Method
Deep Reinforcement Learning-Based Image Captioning with Embedding Reward
Ren, Z., Wang, X., Zhang, N., Lv, X., & Li, L.-J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.128 · Cited by 338 · cited 2× · see citing works
“Many image captioning methods [26, 39, 61, 112, 119, 126, 135, 144, 149, 151, 156] use the dataset in their experiments.”
Method
Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval
Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., & Manning, C. D. (2015) · Workshop on Vision and Language (VL) · doi:10.18653/v1/w15-2812 · Cited by 336 · cited 1× · see citing works
“PICE. SPICE (Semantic Propositional Image Caption Evaluation) [3] is a new caption evaluation metric based on semantic concept. It is based on a graph-based semantic representation called scene-graph [63, 123]. This graph can extract the information of different objects, attributes and their relationships from the ima…”
Method
Collective Generation of Natural Image Descriptions
Кузнецова, П., Ordóñez, V., Berg, A., Berg, T. L., & Choi, Y. (2012) · Cited by 320 · see citing works
The IAPR Benchmark : A New Evaluation Resource for Visual Information Systems
Grubinger, M. (2006) · RWTH Publications (RWTH Aachen) · Cited by 303 · see citing works
Image Captioning with Deep Bidirectional LSTMs
Wang, C., Yang, H., Bartz, C., & Meinel, C. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964299 · Cited by 294 · cited 4× · see citing works
“[21, 59, 61, 144, 150, 152] have performed experiments using the dataset.”
MethodInfluential
StyleNet: Generating Attractive Visual Captions with Styles
Gan, C., Gan, Z., He, X., Gao, J., & Deng, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.108 · Cited by 287 · cited 2× · see citing works
“Gan et al. [39] proposed a novel image captioning system called StyleNet.”
BackgroundMethod
Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections
Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., & Lazebnik, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_35 · Cited by 286 · cited 1× · see citing works
“The captions for the query image are selected from this captions pool [47, 55, 108, 130].”
Background
Image Description using Visual Dependency Representations
Elliott, D., & Keller, F. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1128 · Cited by 282 · cited 2× · see citing works
“mmatically correct captions. However, templates are predefined and cannot generate variable-length captions. Moreover, later on, parsing based language models have been introduced in image captioning [2, 32, 76, 77, 101] which are more powerful than fixed template-based methods. Therefore, in this paper, we do not foc…”
Method
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R. J., Saenko, K., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.8 · Cited by 280 · cited 1× · see citing works
“Anne et al. [6] proposed a Deep Compositional Captioner (DCC) that can represent the unseen objects in generated captions.”
Background
T ree T alk : Composition and Compression of Trees for Image Descriptions
Kuznetsova, P., Ordóñez, V., Berg, T. L., & Choi, Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00188 · Cited by 268 · cited 1× · see citing works
“Moreover, later on, parsing-based language models have been introduced in image captioning [2, 32, 76, 77, 101], which are more powerful than fixed template-based methods.”
Background
Bottom-Up and Top-Down Attention for Image Captioning and VQA.Preprint
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2017) · arXiv (Cornell University) · Cited by 252 · see citing works
A survey on automatic image caption generation
Bai, S., & An, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.080 · Cited by 243 · cited 1× · see citing works
“So far, only three survey papers [8, 13, 75] have been published on this research topic.”
Background
Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
Shetty, R., Rohrbach, M., Hendricks, L. A., Fritz, M., & Schiele, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.445 · Cited by 231 · see citing works
Review Networks for Caption GenerationPreprint
Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07912 · Cited by 230 · cited 4× · see citing works
“[151] proposed a semanticconcept-based image captioning method.”
MethodInfluential
Areas of Attention for Image Captioning
Pedersoli, M., Lucas, T., Schmid, C., & Verbeek, J. J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.140 · Cited by 215 · cited 5× · see citing works
“[112] proposed an area-based attention mechanism for image captioning.”
BackgroundMethodResultInfluential
SentiCap: Generating Image Descriptions with Sentiments
Mathews, A., Xie, L., & He, X. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10475 · Cited by 215 · cited 3× · see citing works
“Jin et al. 2015 [61] Flickr 8K/30K, MS COCO BLEU, METEOR, ROUGE, CIDEr Wu et al. 2016 [151] MS COCO BLEU, METEOR, CIDEr Sugano et at. 2016 [129] MS COCO BLEU, METEOR, ROUGE, CIDEr Mathews et al. 2016 [97] MS COCO, SentiCap BLEU, METEOR, ROUGE, CIDEr Wang et al. 2016 [144] Flickr 8K/30K, MS COCO BLEU, R@K Johnson et al…”
BackgroundMethodInfluential
Attention Correctness in Neural Image Captioning
Liu, C., Mao, J., Sha, F., & Yuille, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11197 · Cited by 210 · cited 1× · see citing works
“Liu et al. [84] proposed a method for neural image captioning.”
Method
Language Models for Image Captioning: The Quirks and What Works
Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015) · doi:10.3115/v1/p15-2017 · Cited by 207 · cited 1× · see citing works
“However, in terms of performance there exists a large gap between the evaluation of the metrics and human judgement of evaluation [20, 30, 74].”
Background
Learning Attributes Equals Multi-Source Domain Generalization
Gan, C., Yang, T., & Gong, B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.17 · Cited by 190 · cited 2× · see citing works
“However, Wu et al. Recent semantic concept based image captioning methods [150, 156] applied semantic-concept-detection process [40]to obtain explicit semantic concepts.”
Method
Captioning Images with Diverse Objects
Venugopalan, S., Hendricks, L. A., Rohrbach, M., Mooney, R. J., Darrell, T., & Saenko, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.130 · Cited by 178 · cited 3× · see citing works
“recognize the unseen objects from unpaired images and incorporate them with LSTM to generate captions. Generating captions for the unseen images is a challenging research problem. Venugopalan et al. [140] introduced a Novel Object Captioner (NOC) for generating captions for unseen objects in the image. They used exter…”
BackgroundInfluential
Dense Captioning with Joint Inference and Visual Context
Yang, L., Tang, K., Yang, J., & Li, L.-J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.214 · Cited by 174 · cited 1× · see citing works
“Yang et al. [153] proposed another dense captioning method.”
Method
Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
Park, C. C., Kim, B., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.681 · Cited by 165 · cited 1× · see citing works
“Park et al. [111] introduced a different type of attention-based image captioning method.”
Method
Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner
Chen, T.-H., Liao, Y.-H., Chuang, C.-Y., Hsu, W.-T., Fu, J., & Sun, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.64 · Cited by 158 · see citing works
Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
Yao, T., Pan, Y., Li, Y., & Mei, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.559 · Cited by 157 · cited 1× · see citing works
“Yao et al. [154] proposed a copying mechanism to generate description for novel objects.”
Background
An Empirical Study of Language CNN for Image Captioning
Gu, J., Wang, G., Cai, J., & Chen, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.138 · Cited by 156 · cited 1× · see citing works
“Inspired by the above success of CNNs in sequence learning tasks, Gu et al. [51] proposed a CNN language model-based image captioning method.”
Method
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.291 · Cited by 147 · cited 1× · see citing works
“The top-down paradigms [23, 31, 65, 93, 94, 142, 152] are used for extracting visual features first and then convert them into words.”
Method
A Game-Theoretic Approach to Generating Spatial Descriptions
Golland, D., Liang, P., & Klein, D. (2010) · Cited by 132 · see citing works
Training Object Class Detectors from Eye Tracking Data
Papadopoulos, D. P., Clarke, A. D. F., Keller, F., & Ferrari, V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_24 · Cited by 131 · cited 1× · see citing works
“Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160].”
Background
Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
Wang, Y., Lin, Z., Shen, X., Cohen, S. D., & Cottrell, G. W. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.780 · Cited by 113 · cited 1× · see citing works
“Wang et al. [148] proposed a method that locates the objects and their interactions first and then identifies and extracts the relevant attributes to generate image captions.”
Method
Rich Image Captioning in the Wild
Tran, K., He, X., Zhang, L., & Sun, J. Z. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.61 · Cited by 110 · see citing works
METEOR, M-BLEU and M-TER
Agarwal, A., & Lavie, A. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626406 · Cited by 109 · cited 1× · see citing works
“nt. There are certain key entities such as celebrities and landmarks that are out of their scope. The generated captions of these methods are evaluated on automatic metrics such as BLEU [110], METEOR [1], and CIDEr [139]. These evaluation metrics have already shown good results on these methods. However, in terms of p…”
Method
Automatic Concept Discovery from Parallel Text and Visual Corpora
Sun, C., Gan, C., & Nevatia, R. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.298 · Cited by 109 · cited 1× · see citing works
“The captions for the query image are selected from these captions pool [47, 55, 108, 130].”
Background
The use of spatial relations in referring expression generation
Viethen, J., & Dale, R. (2008) · doi:10.3115/1708322.1708334 · Cited by 105 · cited 1× · see citing works
“edding space. Mao et al. [92] proposed a special type of text generation method for images. This method can generate a description for an specific object or region that is called referring expression [37, 46, 68, 102, 103, 136, 141]. Using this expression it can then infer the object or region which is being described…”
Method
Framing image description as a ranking task
HodoshMicah, YoungPeter, & HockenmaierJulia (2013) · doi:10.5555/2566972.2566993 · Cited by 98 · see citing works
Aligning where to see and what to tell: image caption with region-based attention and scene factorizationPreprint
Jin, J., Fu, K., Cui, R., Sha, F., & Zhang, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06272 · Cited by 97 · cited 3× · see citing works
“[21, 59, 61, 144, 150, 152] have performed experiments using the dataset.”
Method
Active Visual Segmentation
Mishra, A. K., Aloimonos, Y., Cheong, L. F., & Kassim, A. A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.171 · Cited by 91 · cited 1× · see citing works
“Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160].”
Background
Generating Image Descriptions Using Dependency Relational Patterns
Aker, A., & Gaizauskas, R. (2010) · Cited by 85 · see citing works
Paying Attention to Descriptions Generated by Image Captioning Models
Tavakoli, H. R., Shetty, R., Borji, A., & Laaksonen, J. T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.272 · Cited by 82 · see citing works
Learning Distributions over Logical Forms for Referring Expression Generation
FitzGerald, N., Artzi, Y., & Zettlemoyer, L. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1197 · Cited by 81 · cited 1× · see citing works
“edding space. Mao et al. [92] proposed a special type of text generation method for images. This method can generate a description for an specific object or region that is called referring expression [37, 46, 68, 102, 103, 136, 141]. Using this expression it can then infer the object or region which is being described…”
Method
Building a semantically transparent corpus for the generation of referring expressions
van Deemter, K., van der Sluis, I., & Gatt, A. (2006) · doi:10.3115/1706269.1706296 · Cited by 78 · see citing works
Attention Correctness in Neural Image CaptioningPreprint
Liu, C., Mao, J., Sha, F., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09553 · Cited by 73 · cited 1× · see citing works
“Liu et al. [84] proposed a method for neural image captioning.”
Method
Generating Expressions that Refer to Visible Objects
Mitchell, M., Hltcoe, J. H., van Deemter, K., & Reiter, E. (2013) · Aberdeen University Research Archive (Aberdeen University) · Cited by 64 · see citing works
A parallel-fusion RNN-LSTM architecture for image caption generation
Wang, M., Song, L., Yang, X., & Luo, C. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7533201 · Cited by 58 · cited 1× · see citing works
“Wang et al. [146] proposed a parallel-fusion RNN-LSTM architecture for image caption generation.”
Background
Eye tracking assisted extraction of attentionally important objects from videos
Karthikeyan, S., Ngo, T., Eckstein, M. P., & Manjunath, B. S. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298944 · Cited by 54 · cited 1× · see citing works
“Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67], and holistic scene understanding [158, 160].”
Background
Joint Image-Text Representation by Gaussian Visual-Semantic Embedding
Ren, Z., Jin, H., Lin, Z., Chen, F., & Yuille, A. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967212 · Cited by 51 · cited 1× · see citing works
“Visual semantic embedding [117, 118] is used to compute the actual reward value in predicting the correct word.”
Method
Exploring the role of gaze behavior and object detection in scene understanding
Yun, K., Peng, Y., Samaras, D., Zelinsky, G. J., & Berg, T. L. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00917 · Cited by 50 · cited 1× · see citing works
“hus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160]. However, Sugano et al. [129] claimed that gaze information has not yet been integrated in image captioning…”
Method
Prediction of search targets from fixations in open-world settings
Sattar, H., Muller, S., Fritz, M., & Bulling, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298700 · Cited by 50 · cited 1× · see citing works
“ption of humans about a scene. Human gaze can identify the important locations of objects in an image. Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160]. How…”
Background
Natural reference to objects in a visual domain
Mitchell, M., van Deemter, K., & Reiter, E. (2010) · International Conference on Natural Language Generation · Cited by 44 · see citing works
Prediction of Search Targets From Fixations in Open-World Settings
Sattar, H., Müller, S., Fritz, M., & Bulling, A. (2015) · Cited by 39 · see citing works
Text Generation Based on Generative Adversarial Nets with Latent Variables
Wang, H., Qin, Z., & Wan, T. R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93037-4_8 · Cited by 37 · see citing works
Describing images by feeding LSTM with structural words
Ma, S., & Han, Y. (2016) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2016.7552883 · Cited by 34 · cited 1× · see citing works
“Maet al. [90] proposed another compositional network-based image captioning method.”
Rich Image Captioning in the WildPreprint
Tran, K., He, X., Zhang, L., Sun, J., Carapcea, C., Thrasher, C., Buehler, C., & Sienkiewicz, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.09016 · Cited by 34 · cited 2× · see citing works
“However, Tran et al. [135] introduced a different image captioning method.”
Method
From Where and How to What We See
Karthikeyan, S., Jagadeesh, V., Shenoy, R., Ecksteinz, M., & Manjunath, B. S. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.83 · Cited by 33 · cited 1× · see citing works
“nt locations of objects in an image. Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160]. However, Sugano et al. [129] claimed that gaze information has not ye…”
Method
Simple Image Description Generator via a Linear Phrase-Based ApproachPreprint
Lebret, R., Pinheiro, P. O., & Collobert, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.8419 · Cited by 28 · cited 1× · see citing works
“In bottom-up approaches, [32, 34, 74, 76, 78, 80], visual concepts (e.”
Understanding scene understanding
Zelinsky, G. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00954 · Cited by 27 · cited 1× · see citing works
“Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling [18, 35, 109, 122, 124], object localization [100] or recognition [67] and holistic scene understanding [158, 160].”
Background
BLEU in characters: towards automatic MT evaluation in languages without word delimiters
Denoual, E., & Lepage, Y. (2004) · Cited by 25 · see citing works
Learning-based Composite Metrics for Improved Caption Evaluation
Sharif, N., White, L., Bennamoun, M., & Shah, S. A. A. (2018) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p18-3003 · Cited by 23 · cited 1× · see citing works
“e 0. Acceptance Date: October 2018. 0:26 Hossain et al. aspects. However, commonly used evaluation metrics consider only some specific features (e.g., lexical or semantic) of languages. Sharif et al. [125] proposed learning-based composite metrics for evaluation of image captions. The composite metric incorporates a s…”
Background
Actor-citic agorithms
Konda, V. R., & Tsitsiklis, J. N. (1999) · neural information processing systems · Cited by 20 · see citing works
A survey of evolution of image captioning techniques
Kumar, A., & Goel, S. (2017) · International Journal of Hybrid Intelligent Systems · doi:10.3233/his-170246 · Cited by 18 · cited 1× · see citing works
“d on image captioning with deep machine learning being popularly used. Deep learning algorithms can handle complexities and challenges of image captioning quite well. So far, only three survey papers [8, 13, 75] have been published on this research topic. Although the papers have presented a good literature survey of…”
Background
Image Captioning with an Intermediate Attributes Layer.
Wu, Q., Shen, C., van den Hengel, A., Liu, L., & Dick, A. (2015) · Cited by 17 · see citing works
Proceedings of the 5th Workshop on Vision and Language
Pezzelle, S., Ravi Shekhar, R. S., & Bernardi, R. (2016) · doi:10.18653/v1/w16-32 · Cited by 17 · see citing works
Skeleton Key: Image Captioning by Skeleton-Attribute DecompositionPreprint
Wang, Y., Lin, Z., Shen, X., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06972 · Cited by 16 · cited 1× · see citing works
“Wang et al. [148] proposed a method that locates the objects and their interactions first and then identifies and extracts the relevant attributes to generate image captions.”
Method
Describing Images with Hierarchical Concepts and Object Class Localization
Han, Y., & Li, G. (2015) · ACM International Conference on Multimedia Retrieval (ICMR) · doi:10.1145/2671188.2749290 · Cited by 8 · cited 1× · see citing works
“It also uses a multi-task method similar to multiple instance learning method [33], and multi-layer optimization method [52] to generate structural words.”
Method
Dense Captioning with Joint Inference and Visual ContextPreprint
Yang, L., Tang, K., Yang, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06949 · Cited by 8 · cited 1× · see citing works
“Yang et al. [153] proposed another dense captioning method.”
Method

Flagged first, then by citations

Sorted by citations

A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836 · see citing works
“The use of deep learning for image captioning has been surveyed very recently [230], [231], and tasks that generate text given textual inputs are discussed in Sections IV-E–IV-G.”
Background
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · see citing works
“One approach to providing grounding is to train distributional models on corpora augmented with perceptual data, such as photos (Hossain et al., 2019) or other modalities (Kiela and Clark, 2015; Kiela et al., 2015).”
Background
Convolutional neural networks in medical image understanding: a survey
Sarvamangala, D. R., & Kulkarni, R. V. (2021) · Evolutionary Intelligence · doi:10.1007/s12065-020-00540-3 · Cited by 929 · see citing works
“Researchers are also working on image captioning (textual representations of the image) [29].”
Background
Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
“video [179] while text-to-image generation aims to generate a realistic image that matches the given text description [180].”
Background
Learning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting
Guo, S., Lin, Y., Wan, H., Li, X., & Cong, G. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3056502 · Cited by 774 · see citing works
“It has been widely used and achieved great successes in various tasks such as natural language processing (NLP) [39], [40], image caption [41] and speech recognition [42].”
Background
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625 · see citing works
“If we take a broader view, many works are using cross-modal deep generative models (such as GAN) in data synthesis, such as from video to audio [266], from text to image and vice versa [267, 268].”
Background
A novel genetic LSTM model for wind power forecast
Shahid, F., Zameer, A., & Muneeb, M. (2021) · Energy · doi:10.1016/j.energy.2021.120069 · Cited by 533 · see citing works
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 522 · see citing works
A review of deep learning with special emphasis on architectures, applications and recent trends
Sengupta, S., Basak, S., Saikia, P., Paul, S., Tsalavoutis, V. A., Atiah, F. D., Ravi, V., & Peters, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105596 · Cited by 468 · see citing works
Thin MobileNet: An Enhanced MobileNet Architecture
Sinha, D., & El‐Sharkawy, M. (2019) · IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon47517.2019.8993089 · Cited by 424 · see citing works
“In real-time applications such as image classiﬁcation [8], captioning [9], object detection [10,23,33] and semantic segmentation [34] in autonomous driving, the inference time and accuracy are very important factors to one's safety.”
Background
The four dimensions of social network analysis: An overview of research methods, applications, and software tools
Camacho, D., Panizo-LLedot, Á., Bello-Orgaz, G., González-Pardo, A., & Cambria, E. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.05.009 · Cited by 339 · see citing works
Visual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 345 · see citing works
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373 · see citing works
“Compared to previous surveys on image captioning [16], [17], [18], [21], [22], we provide a comprehensive and updated view on deep learning-based generative captioning models.”
Background
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304 · see citing works
“StandardCaptioning( Cap )loss[45] also aims to predict the next token given previous tokens and images.”
Computer vision applications in construction safety assurance
Fang, W., Ding, L., Love, P. E. D., Luo, H., Li, H., Pena-Mora, F. A., Zhong, B., & Zhou, C. (2019) · Automation in Construction · doi:10.1016/j.autcon.2019.103013 · Cited by 290 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286 · see citing works
Vision Transformers in medical computer vision—A contemplative retrospection
Parvaiz, A., Khalid, M. A., Zafar, R., Ameer, H., Ali, M. U., & Fraz, M. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106126 · Cited by 288 · see citing works
Relation between prognostics predictor evaluation metrics and local interpretability SHAP values
Baptista, M. L., Goebel, K., & Henriques, E. M. P. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103667 · Cited by 277 · see citing works
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257 · see citing works
“Beyond text-to-text generation tasks, recent years have witnessed a growing interest in visuallanguage (VL) generation tasks, such as describing visual scenes [46], and answering visual-related questions [81].”
Background
Medical deep learning—A systematic meta-review
Egger, J., Gsaxner, C., Pepe, A., Pomykala, K. L., Jonske, F., Kurz, M., Li, J., & Kleesiek, J. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106874 · Cited by 270 · see citing works
“Object detection [75–77] Image segmentation [ 78 , 79 ] Face recognition [80–82] Action/motion recognition [ 83 , 84 ] Biometric recognition [ 85 , 86 ] Image super-resolution [87] Image captioning [88] Data augmentation [89] Generative adversarial networks [90]”
Improved Few-Shot Visual Classification
Bateni, P., Goyal, R., Masrani, V., Wood, F., & Sigal, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01450 · Cited by 248 · see citing works
Machine Learning for Clinical Outcome Prediction
Shamout, F. E., Zhu, T.-T., & Clifton, D. (2020) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3007816 · Cited by 210 · see citing works
Methods for Pruning Deep Neural Networks
Vadera, S., & Ameen, S. (2022) · IEEE Access · doi:10.1109/access.2022.3182659 · Cited by 229 · see citing works
“There are many different types of deep learning networks, including convolutional neural networks (CNNs), recurrent neural networks and recursive neural networks [75,139].”
Background
Deep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction
Nasser, M., & Yusof, U. K. (2023) · Diagnostics · doi:10.3390/diagnostics13010161 · Cited by 227 · see citing works
“To mitigate the issue of the vanishing and exploding gradients of RNN, new models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were introduced [36].”
MethodInfluential
Artificial intelligence in drug discovery: applications and techniques
Deng, J., Yang, Z., Ojima, I., Samaras, D., & Wang, F. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab430 · Cited by 236 · see citing works
Using AI and Social Media Multimodal Content for Disaster Response and Management: Opportunities, Challenges, and Future Directions
Imran, M., Ofli, F., Caragea, D., & Torralba, A. B. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102261 · Cited by 227 · see citing works
Adversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 214 · see citing works
“T2I can be seen as the inverse of image captioning [15], where the input is an image and the output is a textual description of that image.”
Background
Dynamic Graph Enhanced Contrastive Learning for Chest X-Ray Report Generation
Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., & Chang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00325 · Cited by 206 · see citing works
Designing ECG monitoring healthcare system with federated transfer learning and explainable AI
Raza, A., Tran, K. P., Koehl, L., & Li, S. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107763 · Cited by 205 · see citing works
“However, thanks to the internet, IoT devices and the increasing computational power, nowadays we can see deep learning revolutionizing nearly every ﬁeld, including healthcare [5], economics [6], manufacturing [7], agriculture [8], and military [9].”
Background
Evaluating progress in automatic chest X-ray radiology report generation
Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E. P., Fonseca, E. K. U. N., Lee, H. M. H., Abad, Z. S. H., Ng, A. Y., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100802 · Cited by 200 · see citing works
“Most prior works have relied on a set of automated metrics inspired by similar setups in natural language generation, where radiology report text is treated as generic text 21 .”
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 205 · see citing works
“However, instead of a pairwise comparison like in [58], we have organized our paper to feature a separate section for each method category and follow the same order of category in our discussion (section 5).”
BackgroundMethod
Scalable Deep Learning on Distributed Infrastructures
Mayer, R., & Jacobsen, H. (2019) · ACM Computing Surveys · doi:10.1145/3363554 · Cited by 234 · see citing works
“, automatically colorize grayscale images) [75], image captioning [68] (i.”
Background
The future of coastal monitoring through satellite remote sensing
Vitousek, S., Buscombe, D., Vos, K., Barnard, P. L., Ritchie, A. C., & Warrick, J. A. (2022) · Cambridge Prisms Coastal Futures · doi:10.1017/cft.2022.4 · Cited by 192 · see citing works
Deep Learning (CNN) and Transfer Learning: A Review
Gupta, J., Pathak, S. K., & Kumar, G. (2022) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2273/1/012029 · Cited by 224 · see citing works
“CNN's coverage of DL has made it a household name [10], [11].”
Background
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182 · see citing works
Translation between Molecules and Natural Language
Edwards, C., Lai, T., Ros, K., Honke, G., Cho, K., & Ji, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.26 · Cited by 225 · see citing works
Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01363 · Cited by 166 · see citing works
“These models have demonstrated profound capabilities in various applications and significantly enhance the performance in image reasoning tasks [5, 18, 20, 30, 31, 36, 38, 42, 47].”
Background
Description Generation for Remote Sensing Images Using Attribute Attention Mechanism
Zhang, X., Wang, X., Tang, X., Zhou, H., & Li, C. (2019) · Remote Sensing · doi:10.3390/rs11060612 · Cited by 146 · see citing works
“There are three mai categories of the existing image captioning m thods [28]: template, retrieval, and encoder-decoderbased im ge capti ni g.”
Background
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K. I., Yu, L., Shan, Q., Nichols, J. W., Wu, J., Fleizach, C., et al. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 145 · see citing works
“To describe content in pictures, prior work used deep learning models to generate natural language descriptions of images [44, 46], and some accessibility improvement research has also leveraged crowdsourcing to generate image captions [35, 39, 40].”
Method
Comprehensive Image Captioning via Scene Graph Decomposition
Zhong, Y., Wang, L., Chen, J., Yu, D., & Li, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_13 · Cited by 137 · see citing works
Video Processing Using Deep Learning Techniques: A Systematic Literature Review
Sharma, V., Gupta, M., Kumar, A., & Mishra, D. (2021) · IEEE Access · doi:10.1109/access.2021.3118541 · Cited by 143 · see citing works
“Another motivation is that in the past ten years, so many review papers are present for deep learning for image processing[17][18]; similarly, deep learning techniques for video processing are needed.”
Background
PLA: Language-Driven Open-Vocabulary 3D Scene Understanding
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., & Qi, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00677 · Cited by 131 · see citing works
“As image captioning is a fundamental task in VL research area [18], various foundation models [40, 1, 30] trained with massive samples are readily available for solving this task.”
Background
Meta Graph Transformer: A Novel Framework for Spatial–Temporal Traffic Prediction
Ye, X., Fang, S., Sun, F., Zhang, C., & Xiang, S. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.12.033 · Cited by 123 · see citing works
Levels of explainable artificial intelligence for human-aligned conversational explanations
Dazeley, R., Vamplew, P. W., Foale, C., Young, C., Aryal, S., & Cruz, F. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103525 · Cited by 122 · see citing works
“For example, Deep Reinforcement Learning [116–119] is a class of algorithms that essentially utilise a combination of two distinct algorithms Reinforcement Learning and Deep Neural Networks.”
Background
Physical Adversarial Attack Meets Computer Vision: A Decade Survey
Wei, H., Tang, H., Jia, X., Wang, Z., Yu, H., Li, Z., Satoh, S., Gool, L. V., & Wang, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3430860 · Cited by 115 · see citing works
“7 for illustrations.”
Deep Learning and Machine Learning with Grid Search to Predict Later Occurrence of Breast Cancer Metastasis Using Clinical Data
Jiang, X., & Xu, C. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11195772 · Cited by 120 · see citing works
SD-RSIC: Summarization-Driven Deep Remote Sensing Image Captioning
Sümbül, G., Nayak, S., & Demir, B. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3031111 · Cited by 110 · see citing works
A Systematic Review of Urban Navigation Systems for Visually Impaired People
Eltaher, F., Taha, A., Courtney, J., & McKeever, S. (2021) · Sensors · doi:10.3390/s21093103 · Cited by 110 · see citing works
“While there are different datasets available for this task [129], none of them were captured from the BVIP perspective.”
Background
Visual to Text: Survey of Image and Video Captioning
Li, S., Tao, Z., Li, K., & Fu, Y. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2892755 · Cited by 108 · see citing works
Improved Automatic Summarization of Subroutines via Attention to File Context
Haque, S., LeClair, A., Wu, L., & McMillan, C. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387449 · Cited by 108 · see citing works
“) in the same input sequence, while in other research areas such as image captioning [20] it has been shown that better results are achieved when different input types are processed separately.”
Background
Automatic Image Captioning Based on ResNet50 and LSTM with Soft Attention
Chu, Y., Xiao, Y., Yu, L., Sergei, M., & Wang, Z. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8909458 · Cited by 108 · see citing works
“Much research effort has been devoted to automatic image captioning, and it can be categorized into templatebased image captioning, retrieval-based image captioning, and novel image caption generation [5].”
Background
Alpha-CLIP: A CLIP Model Focusing on Wherever you Want
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01237 · Cited by 119 · see citing works
“While CLIP captures the content of the entire image, it is also crucial to focus on the regions of interest to enable a finer understanding [14, 18, 21, 37, 48, 72] and controllable content generation [22, 33, 46, 55].”
Automatic captioning for medical imaging (MIC): a rapid review of literature
Beddiar, D. R., Oussalah, M., & Seppänen, T. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10270-w · Cited by 103 · see citing works
“This is due, in fact, to both the inherent properties of medical images with their complex anatomy structures and to the fact that the annotations should only focus on clinically important aspects that are relevant to the diagnosis, rather than mapping all objects in the image as in a generic image captioning system (…”
Background
Knowledge and topology: A two layer spatially dependent graph neural networks to identify urban functions with time-series street view image
Zhang, Y., Liu, P., & Biljecki, F. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.03.008 · Cited by 100 · see citing works
“Machine reasoning is used to deconstruct and understand the content of the image, then generate a natural semantic description of the given scene (Hossain et al., 2019).”
Large Language Models versus Natural Language Understanding and Generation
Karanikolas, N., Manga, E., Samaridi, N., Tousidou, E., & Vassilakopoulos, M. (2023) · Proceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics · doi:10.1145/3635059.3635104 · Cited by 102 · see citing works
Machine Learning-Based Intrusion Detection Methods in IoT Systems: A Comprehensive Review
Kikissagbe, B. R., & Adda, M. (2024) · Electronics · doi:10.3390/electronics13183601 · Cited by 127 · see citing works
“These results confirm the potential of SVM as a robust tool for anomaly detection in IoT network security [32].”
Result
From image to language: A critical analysis of Visual Question Answering (VQA) approaches, challenges, and opportunities
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 100 · see citing works
“A similar mechanism is employed in VQA following the success of attention in standard computer vision tasks like object recognition [228] and multimodal tasks like image captioning [229, 8].”
BackgroundMethodInfluential
Fully automated carbonate petrography using deep convolutional neural networks
Koeshidayatullah, A., Morsilli, M., Lehrmann, D. J., Al-Ramadan, K. A., & Payne, J. L. (2020) · Marine and Petroleum Geology · doi:10.1016/j.marpetgeo.2020.104687 · Cited by 97 · see citing works
Orderless Recurrent Models for Multi-Label Classification
Yazici, V. O., González-García, A., Arnau Ramisa, A., Twardowski, B., & van de Weijer, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01345 · Cited by 95 · see citing works
A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance
Domingos, L. C. F., Santos, P. E., Skelton, P. S. M., Brinkworth, R. S. A., & Sammut, K. (2022) · Sensors · doi:10.3390/s22062181 · Cited by 93 · see citing works
“There have been significant recent developments in Deep Learning (DL) methods, mostly pushed by object detection and classification in images [45,46] and visual question answering [47].”
Background
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299 · see citing works
Image captioning with transformer and knowledge graph
Zhang, Y., Shi, X., Mi, S., & Yang, X. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.12.020 · Cited by 87 · see citing works
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 84 · see citing works
Incorporating attentive multi-scale context information for image captioning
Prudviraj, J., Yenduri, S., & Mohan, C. K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11895-9 · Cited by 82 · see citing works
“Several pioneering works [20, 23, 39, 55, 59, 64, 65] rely on such combination of CNN+RNN encoder-decoder approach.”
BackgroundMethod
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 80 · see citing works
“education, digital libraries, and web searching [95].”
A survey on automatic generation of medical imaging reports based on deep learning
Pang, T., Li, P., & Zhao, L. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01113-y · Cited by 80 · see citing works
“The automatic generation of diagnostic reports is inspired by image captioning [2], which combines computer vison (CV) and natural language processing (NLP) to provide a comprehensive understanding of medical images.”
Method
Deep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 78 · see citing works
Towards goal-oriented semantic signal processing: Applications and future challenges
Kalfa, M., Gok, M., Atalık, A., Tegin, B., Duman, T. M., & Arıkan, O. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2021.103134 · Cited by 74 · see citing works
“A comprehensive survey of image and video captioning models can be found in [110] and [103], respectively.”
Background
Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation
Yang, Y. Q., Yu, J., Zhang, J., Han, W., Jiang, H., & Huang, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3122542 · Cited by 73 · see citing works
“The earliest works generate image captions with template-based or retrieval-based methods [11].”
Background
Evaluation of artificial intelligence model for crowding categorization and extraction diagnosis using intraoral photographs
Ryu, J., Kim, Y.-H., Kim, T., & Jung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-32514-7 · Cited by 73 · see citing works
Digital Technology and AI for Smart Sustainable Cities in the Global South: A Critical Review of Literature and Case Studies
Das, D. K. (2025) · Urban Science · doi:10.3390/urbansci9030072 · Cited by 89 · see citing works
“For example, DL applications such as image and video recognition are essential for urban security and surveillance, helping to identify objects, faces, and scenes in real time [119].”
Enhancing Few-Shot Image Classification with Unlabelled Examples
Bateni, P., Barber, J., van de Meent, J.-W., & Wood, F. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00166 · Cited by 69 · see citing works
A Systematic Literature Review on Image Captioning
Staniūtė, R., & Šešok, D. (2019) · Applied Sciences · doi:10.3390/app9102024 · Cited by 68 · see citing works
“There are only a few SLRs that have been conducted for image captioning until now [6–9], though with such fast progress and increasing popularity in this ﬁeld we ﬁnd it necessary that they continue to be undertaken.”
BackgroundMethod
Knowledge Graphs for Multi-modal Learning: Survey and Perspective
Chen, Z., Zhang, Y., Fang, Y., Geng, Y., Guo, L., Chen, J., Liu, X., Pan, J. Z., Zhang, N., Chen, H., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103124 · Cited by 72 · see citing works
Text to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 67 · see citing works
“[18] present a comprehensive survey of the topic.”
BackgroundSelf-citation
From vision to text: A comprehensive review of natural image captioning in medical diagnosis and radiology report generation
Reale-Nosei, G., Amador-Domínguez, E., & Serrano, E. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103264 · Cited by 68 · see citing works
A Multi-Level Attention Model for Remote Sensing Image Captions
Li, Y., Fang, S., Jiao, L., Liu, R., & Shang, R. (2020) · Remote Sensing · doi:10.3390/rs12060939 · Cited by 65 · see citing works
Neural attention for image captioning: review of outstanding methods
Zohourianshahzadi, Z., & Kalita, J. K. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10092-2 · Cited by 61 · see citing works
“Recently several high-quality surveys on image captioning with deep neural networks (Hossain et al. 2019; Liu et al. 2019b) and other techniques such as template generation and slot filling (Bai and An 2018; Sharma et al.”
Background
VaBUS: Edge-Cloud Real-Time Video Analytics via Background Understanding and Subtraction
Wang, H., Li, Q., Sun, H., Chen, Z., Hao, Y., Peng, J., Yuan, Z., Fu, J., & Jiang, Y. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221995 · Cited by 60 · see citing works
“…of deep learning models have been proposed for various vision applications, including image classiﬁcation [51], instance segmentation [52], object detection [53], face recognition [54], image captioning [55], etc. Recent works show that it is inefﬁcient to perform inference for every video frame.”
Background
A Joint-Training Two-Stage Method For Remote Sensing Image Captioning
Ye, X., Wang, S., Gu, Y., Wang, J., Wang, R., Hou, B., Giunchiglia, F., & Jiao, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3224244 · Cited by 60 · see citing works
“Therefore, RSI captioning has attracted increasing attention from researchers [1], [2], [3], [4].”
Background
Explainable reinforcement learning for broad-XAI: a conceptual framework and survey
Dazeley, R., Vamplew, P. W., & Cruz, F. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08423-1 · Cited by 68 · see citing works
“Hence, function approximation techniques such as Neural Networks are frequently utilised, giving rise to the ﬁeld of Deep RL (DRL) [163, 144, 15, 66, 109, 226].”
Method
Dynamic Attention Guided Multi-Trajectory Analysis for Single Object Tracking
Wang, X., Chen, Z., Tang, J., Luo, B., Wang, Y., Tian, Y., & Wu, F. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3056684 · Cited by 56 · see citing works
ArchGPT: harnessing large language models for supporting renovation and conservation of traditional architectural heritage
Zhang, J., Xiang, R., Kuang, Z., Wang, B., & Li, Y. (2024) · Heritage Science · doi:10.1186/s40494-024-01334-x · Cited by 56 · see citing works
“For the task planning aspect, we will define necessary task steps based on prior knowledge (e.g., when an image is inputted, the Image Caption model [18] must be called to supplement LLM’s understanding of the image), to provide essential inputs for the final response (refer to "Task parsing"…”
Method
A trend graph attention network for traffic prediction
Wang, C., Tian, R., Hu, J., & Ma, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.12.048 · Cited by 53 · see citing works
Deep learning in fringe projection: A review
Liu, H., Yan, N., Shao, B., Yuan, S., & Zhang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127493 · Cited by 56 · see citing works
Prompt-Based Learning for Unpaired Image Captioning
Zhu, P., Wang, X., Zhu, L., Sun, Z., Zheng, W., Wang, Y., & Chen, C. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3265842 · Cited by 52 · see citing works
A comprehensive survey on image captioning: from handcrafted to deep learning-based techniques, a taxonomy and open research issues
Sharma, H., & Padha, D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10488-2 · Cited by 51 · see citing works
Updated Primer on Generative Artificial Intelligence and Large Language Models in Medical Imaging for Medical Professionals
Kim, K., Cho, K., Jang, R., Kyung, S., Lee, S., Ham, S., Choi, E., Hong, G.-S., & Kim, N. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0818 · Cited by 54 · see citing works
“AI models create relevant and kjronline.org descriptive captions for the visual inputs provided in visual captioning [49].”
Spatial dynamic graph convolutional network for traffic flow forecasting
Li, H., Yang, S., Song, Y., Luo, Y., Li, J., & Zhou, T. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04271-z · Cited by 50 · see citing works
“Attention mechanism has been widely applied in natural language processing (NLP) [32, 33], image recognition [34, 35], protein identification [36, 37], recommended system [38] and etc. Self-attention which emphasizes many-to-many is a variant of Attention mechanism.”
Background
Construction safety inspection with contrastive language-image pre-training (CLIP) image captioning and attention
Tsai, W.-L., Le, P.-L., Ho, W.-F., Chi, N.-W., Lin, J. J., Tang, S., & Hsieh, S. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105863 · Cited by 50 · see citing works
Image captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49 · see citing works
Attention and Long Short-Term Memory Network for Remaining Useful Lifetime Predictions of Turbofan Engine Degradation
da Costa, P. R. D. O., Akçay, A., Zhang, Y., & Kaymak, U. (2023) · International Journal of Prognostics and Health Management · doi:10.36001/ijphm.2019.v10i4.2623 · Cited by 49 · see citing works
“CNNs are notable for being able to extract spatial information from 2D and 3D data (Hossain et al., 2019).”
Background
Mitigating Gender Bias in Captioning Systems
Tang, R., Du, M., Li, Y., Liu, Z., Zou, N., & Hu, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449950 · Cited by 50 · see citing works
Image Captioning Using Inception V3 Transfer Learning Model
Degadwala, S., Vyas, D., Biswas, H., Chakraborty, U., & Saha, S. (2021) · International Conference on Communication and Electronics Systems (ICCES) · doi:10.1109/icces51350.2021.9489111 · Cited by 48 · see citing works
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 128 · see citing works
“We present details for only eight tasks by excluding visual description generation (Aafaq et al., 2018; Hossain et al., 2019) and question answering (Wu et al.”
BackgroundMethodInfluential
Retrieval-augmented Image Captioning
Ramos, R., Elliott, D., & Martins, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.266 · Cited by 47 · see citing works
A Study of ConvNeXt Architectures for Enhanced Image Captioning
Ramos, L., Casas, E., Romero, C., Rivas, F., & Morocho-Cayamcela, M. E. (2024) · IEEE Access · doi:10.1109/access.2024.3356551 · Cited by 47 · see citing works
High-Throughput Vector Similarity Search in Knowledge Graphs
Mohoney, J., Pacaci, A., Chowdhury, S. R., Mousavi, A., Ilyas, I. F., Minhas, U. F., Pound, J., & Rekatsinas, T. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3589777 · Cited by 48 · see citing works
“Modern machine learning models are increasingly used to embed complex data such as images, text, or entities in knowledge graphs into vector representations that retain semantically meaningful information [5, 6, 9, 18, 29].”
Background
Automatic image captioning combining natural language processing and deep neural networks
Rinaldi, A. M., Russo, C., & Tommasino, C. (2023) · Results in Engineering · doi:10.1016/j.rineng.2023.101107 · Cited by 45 · see citing works
Boosting convolutional image captioning with semantic content and visual relationship
Bai, C., Zheng, A., Huang, Y., Pan, X., & Chen, N. (2021) · Displays (Guildford) · doi:10.1016/j.displa.2021.102069 · Cited by 37 · see citing works

A Comparative Study of Various Deep Learning Techniques Based on Automatic Image Captioning
Anurag, & Kumar, N. (2020) · International Journal of Computer Sciences and Engineering
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 205 · see citing works
Image Captioning Techniques: A Review
Al-Jamal, A. Z., Bani-Amer, M. J., & Aljawarneh, S. A. (2022) · International Conference on Engineering & MIS (ICEMIS) · doi:10.1109/icemis56295.2022.9914173 · Cited by 10 · see citing works
Comprehensive Comparative Study on Several Image Captioning Techniques Based on Deep Learning Algorithm
Ningthoujam, C., & Chingtham, T. S. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-16-4244-9_18 · Cited by 1 · see citing works
Textualizing images: A comprehensive review of image captioning using deep learning
Thakur, S., & Bhat, A. (2025) · AIP conference proceedings · doi:10.1063/5.0294151

Authors: Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H.
Affiliations: Murdoch University (AU)
ORCID: Hossain, MD. Zakir
Sohel, Ferdous
Shiratuddin, Mohd Fairuz
Laga, Hamid
Venue: ACM Computing Surveys
Publisher: Association for Computing Machinery (ACM)
Published: 2019-02-04
Volume: 51
Issue: 6
Pages: 1-36
ISSN: 0360-0300 (ISSN-L)
1557-7341
DOI: 10.1145/3295748
OpenAlex: W2896348597
Cited by: 879
References: 166
Citations by year: 2026: 55 · 2025: 113 · 2024: 124 · 2023: 167 · 2022: 153 · 2021: 127 · 2020: 104 · 2019: 36
FWCI: 47.02
Type: journal article
Language: en
Open access: closed
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Computing Surveys (publishedVersion)
Murdoch University Research Portal (submittedVersion)
Funders: Australian Research Council (DE120102960)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords: image captioning, object recognition, sentence generation, evaluation metrics, deep learning architectures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A Comprehensive Survey of Deep Learning for Image Captioning

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks