The DOI matches its Crossref record; 167 of the 167 references it cites were found in a registry or an index.
A Comprehensive Survey of Deep Learning for Image Captioning
Abstract
Generating a description of an image is called image captioning. Image captioning requires recognizing the important objects, their attributes, and their relationships in an image. It also needs to generate syntactically and semantically correct sentences. Deep-learning-based techniques are capable of handling the complexities and challenges of image captioning. In this survey article, we aim to present a comprehensive review of existing deep-learning-based image captioning techniques. We discuss the foundation of the techniques to analyze their performances, strengths, and limitations. We also discuss the datasets and the evaluation metrics popularly used in deep-learning-based automatic image captioning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most prior works have relied on a set of metrics inspired by similar setups in natural language generation, where radiology report text is treated as generic text.22”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [22]
In the citing paper
“If we take a broader view, many works are using cross-modal deep generative models (such as GAN) in data synthesis, such as from video to audio [267], from text to image and vice versa [268,269].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [268]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“The architecture of this method has ResNet [53] and two LSTMs called Skel-LSTM and Attr-LSTM.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“Moreover, recent object detection and classification methods [73, 127] show that deep, hierarchical methods are better at learning than shallower ones.”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5×
“Kiros et al. [69] proposed a method that is built on a Log-Bilinear model and used AlexNet to extract visual features.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM [54] networks are a type of RNN that have special units in addition to standard units.”
BackgroundMethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 4×
“The bottom up attention mechanism uses Faster R-CNN [116] for region proposals that can select salient regions of an image .”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“For example, Convolutional Neural Networks (CNN) [79] are widely used for feature learning, and a classifier such as Softmax is used for classification.”
MethodDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 1×
“d techniques and (2) Deep machine learning based techniques. In traditional machine learning, hand crafted features such as Local Binary Patterns (LBP) [107], Scale-Invariant Feature Transform (SIFT) [87], the Histogram of Oriented Gradients (HOG) [27], and a combination of such features are widely used. In these tech…”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Flagged first, then by citations
Research works printed but not linked: 58
- 10.5555/1626394.1626406 · DOI
- Aker Ahmet, 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 1250--1258
- Bahdanau Dzmitry, 2015, International Conference on Learning Representations (ICLR’15)
- Cristian Bodnar. 2018. Text to image synthesis using generative adversarial networks. arXiv preprint arXiv:1805.00676. Cristian Bodnar. 2018. Text to image synthesis using generative adversarial networks. arXiv preprint arXiv:1805.00676. · arXiv
- Chen Xinlei, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2422--2431
- Cho Kyunghyun, 2014, Dzmitry Bahdanau, and Yoshua Bengio.
- Denoual Etienne, Companion
- Fang Hao, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1473--1482
- 10.5555/1888089.1888092 · DOI
- Fedus William, 2018, Dai
- FitzGerald Nicholas, 2013, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 1914--1925
- Frome Andrea, 2013, Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems. 2121--2129.
- Gehring Jonas, 2016, Dauphin
- Gehring Jonas, 2017, Dauphin
- 10.5555/1870658.1870698 · DOI
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680.
- Gregor Karol, 2015, Proceedings of Machine Learning Research. 1462--1471
- Grubinger Michael, 2006, International Workshop Ontoimage, 5
- Isola Phillip, Proceedings of the IEEE International Conference on Computer Vision (CVPR’17)
- Max Jaderberg Karen Simonyan Andrew Zisserman and Koray Kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025. Max Jaderberg Karen Simonyan Andrew Zisserman and Koray Kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025.
- Jang Eric, 2017, International Conference on Learning Representations (ICLR’17)
- Kazemzadeh Sahar, 2014, Berg
- Kiros Ryan, 2014, Proceedings of the 31st International Conference on Machine Learning (ICML’14)
- Kiros Ryan, Workshop on Neural Information Processing Systems (NIPS’14)
- Konda Vijay R., 2000, Tsitsiklis
- Krizhevsky Alex, 2012, Hinton
- 10.5555/2390524.2390575 · DOI
- Kuznetsova Polina, 2014, TREETALK: Composition and compression of trees for image descriptions.TACL 2, 10
- Lebret RÃl’mi, 2015, Workshop on International Conference on Learning Representations (ICLR’15)
- Li Siming, 2011, Proceedings of the 15th Conference on Computational Natural Language Learning. Association for Computational Linguistics, 220--228
- Lin Chin-Yew, 2004, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 8
- Liu Chenxi, 2017, Yuille
- Maddison Chris J., 2017, International Conference on Learning Representations (ICLR’17)
- Mao Junhua, 2015, International Conference on Learning Representations (ICLR’15)
- Mao Junhua, 2014, Yuille
- Mitchell Margaret, 2012, Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 747--756
- 10.5555/1873738.1873752 · DOI
- Mitchell Margaret, 2013, Kees Van Deemter, and Ehud Reiter
- 10.1145/1273496.1273577 · DOI
- 10.5555/645313.649410 · DOI
- Ordonez Vicente, 2011, Berg
- Ranzato Marc’Aurelio, 2016, International Conference on Learning Representations (ICLR’16)
- Zhou Ren Hailin Jin Zhe Lin Chen Fang and Alan Yuille. 2015. Multi-instance visual-semantic embedding. arXiv preprint arXiv:1512.06963. Zhou Ren Hailin Jin Zhe Lin Chen Fang and Alan Yuille. 2015. Multi-instance visual-semantic embedding. arXiv preprint arXiv:1512.06963. · arXiv
- Vadivel Karthikeyan Shanmuga, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3241--3250
- Simonyan Karen, 2015, International Conference on Learning Representations (ICLR’15)
- Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203. Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203. · arXiv
- Sutskever Ilya, 2014, Le
- van Deemter Kees, Proceedings of the 4th International Natural Language Generation Conference. Association for Computational Linguistics, 130--132
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998--6008.
- 10.5555/1708322.1708334 · DOI
- Wang Qingzhong, 2018, Chan
- Wang Yufei, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17)
- Yang Zhilin, 30th Conference on Neural Image Processing System (NIPS’16)
- Xu Kelvin, 2015, International Conference on Machine Learning. 2048--2057
- Yang Linjie, 2016, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16)
- Yu Lantao, 2017, Proceedings of the 31st AAAI Conference on Artificial Intelligence.
- Matthew, 2014, Zeiler and Rob Fergus
- Zhang Li, 2017, Hospedales
Cited by
Sorted by citations
Items 201–300 of 878
Modeling of Hyperparameter Tuned Deep Learning Model for Automated Image Captioning
Omri, M., Abdel‐Khalek, S., Khalil, E. M., Bouslimi, J., & Joshi, G. P. (2022) · Mathematics · doi:10.3390/math10030288 · Cited by 19
“At the bottom, there is a sentence that describes the image content—the object appearing in the action, the scene, and the image is defined in this sentence [2].”
BackgroundImage Captioning Model Using Part-of-Speech Guidance Module for Description With Diverse Vocabulary
Bae, J.-W., Lee, S.-H., Kim, W.-Y., Seong, J.-H., & Seo, D.-H. (2022) · IEEE Access · doi:10.1109/access.2022.3169781 · Cited by 21
Image captioning based on scene graphs: A survey
Jia, J., Ding, X., Pang, S., Gao, X., Xin, X., Hu, R., & Nie, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120698 · Cited by 20
Multimodal Data Augmentation for Image Captioning using Diffusion Models
Xiao, C., Xu, S. X., & Zhang, K. (2023) · doi:10.1145/3607827.3616839 · Cited by 28
“Encoder-Decoder architecture is one of the most common and effective model frameworks, with Convolutional Neural Networks (CNN) encoders to obtain image features and Recurrent Neural Networks (RNN) for decoding them into natural language [25].”
BackgroundavtmNet:Adaptive Visual-Text Merging Network for Image Captioning
Song, H., Zhu, J., & Jiang, Y. (2020) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2020.106630 · Cited by 19
Encoder-Decoder Architecture for Image Caption Generation
Parikh, H., Sawant, H., Parmar, B., Shah, R., Chapaneri, S., & Jayaswal, D. (2020) · International Conference on Communication System, Computing and IT Applications (CSCITA) · doi:10.1109/cscita47329.2020.9137802 · Cited by 19
“[8] used the method of a CNN and RNN based combined network.”
MethodA Thorough Review on Recent Deep Learning Methodologies for Image CaptioningPreprint
Elhagry, A., & Kadaoui, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.13114 · Cited by 19
“As mentioned in the review paper [9], the authors presented a comprehensive review of the state-of-the-art deep learning-based image captioning techniques by late 2018.”
Aligning First, Then Fusing: A novel weakly supervised multimodal violence detection method
Jin, W., Zhu, L., & Sun, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113709 · Cited by 18
The AI-Medic: an artificial intelligent mentor for trauma surgery
Rojas-Muñoz, E., Couperus, K. S., & Wachs, J. P. (2020) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2020.1835548 · Cited by 20
“These algorithms, however, are trained to generate captions that describe the content of the images using techniques such as template-based image cap-tioning, retrieval-based image captioning, and novel caption generation (Hossain et al. 2019).”
BackgroundA Novel Hybrid Multi-Modal Deep Learning for Detecting Hashtag Incongruity on Social Media
Dadgar, S., & Neshat, M. (2022) · Sensors · doi:10.3390/s22249870 · Cited by 20
“Some application domains that intersect NLP and Computer Vision include image and video captioning [77], document image classification [78] and visual question answering [79].”
BackgroundIntroduction to deep learning and diagnosis in medicine
Oğuz, A., & Ertuğrul, Ö. F. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-96129-5.00003-2 · Cited by 25
Jamil, A., Rehman, S. U., Mahmood, K., Villar, M. G., Prola, T. A., Díez, I. D. L. T., Samad, M. A., & Ashraf, I. (2024) · IEEE Access · doi:10.1109/access.2024.3365528 · Cited by 23
RetractedDomain-specific image captioning: a comprehensive review
Sharma, H., & Padha, D. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00328-6 · Cited by 19
“The VS gathers visual context by leveraging ensemble machine learning and deep learning feature extraction techniques [32].”
InfluentialJack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
Pramanick, S., Han, G., Hou, R., Nag, S., Lim, S.-N., Ballas, N., Wang, Q., Chellappa, R., & Almahairi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01335 · Cited by 35
“…textual instructions, can broadly be categorized into two groups based on their input and output formats: Coarse-level MLLMs: Early attempts of designing MLLMs focused on image-level vision tasks with textual outputs, such as visual question answering [2, 29, 66, 86] and image captioning [24, 26].”
BackgroundAutomatic Myanmar Image Captioning using CNN and LSTM-Based Language Model
Aung, S. P. P., Pa, W. P., & Nwe, T. L. (2020) · Workshop Spoken Language Technologies for Under-resourced Languages · Cited by 18
Metaheuristics Optimization with Deep Learning Enabled Automated Image Captioning System
Duhayyim, M. A., Alazwari, S., Mengash, H. A., Marzouk, R., Alzahrani, J. S., Mahgoub, H., Althukair, F., & Salama, A. S. (2022) · Applied Sciences · doi:10.3390/app12157724 · Cited by 18
“Such origins have images which viewers should analyze themselves [1].”
BackgroundThe bottlenecks of AI: challenges for embedded and real-time research in a data-centric age
Abdelzaher, T., Hu, Y. C., Kara, D., Kimura, T., Misra, A., Ramani, V., Tardieu, O., Wang, T., Wigness, M., & Youssef, A. (2025) · Real-Time Systems · doi:10.1007/s11241-025-09452-w · Cited by 22
Image Captioning with Internal and External Knowledge
Huang, F., Li, Z., Chen, S., Zhang, C., & Ma, H. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411948 · Cited by 17
“Image captioning has recently attracted great attention in the field of artificial intelligence, due to the significant progress of machine learning technologies and the release of a number of large-scale datasets [11] [3] [8].”
BackgroundLin, Z., Zhu, F., Kong, Y., Wang, Q., & Wang, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3185678 · Cited by 17
“The former includes object detection [1]–[4], semantic segmentation [5], [6], and instance segmentation [7], [8], and the latter involves image captioning [9], [10], visual question answering [11], [12], and visual reasoning [13].”
BackgroundSemantic interdisciplinary evaluation of image captioning models
Sirisha, U., & Chandana, B. S. (2022) · Cogent Engineering · doi:10.1080/23311916.2022.2104333 · Cited by 18
“The survey papers (Bai & An, 2018; Choi et al., 2021; Elhagry & Kadaoui, 2021; Hossain et al., 2019; X. Liu et al., 2019; Monshi et al., 2020; Oluwasammi et al., 2021; Pavlopoulos et al., 2021; Staniūtė & Šešok, 2019; Stefanini et al., n.d.; Y. Wang et al., 2019) mainly discussed template,…”
BackgroundMethodAutomatic Defect Description of Railway Track Line Image Based on Dense Captioning
Wei, D., Wei, X., & Jia, L. (2022) · Sensors · doi:10.3390/s22176419 · Cited by 17
“In comparison with other types of image captioning methods [10], this technology focuses attention on image regions containing objects and generates more objective and detailed region-wise captions.”
BackgroundShi, Y., Olsson, L. T., Hoadley, K. A., Calhoun, B. C., Marron, J. S., Geradts, J., Niethammer, M., & Troester, M. A. (2023) · npj Breast Cancer · doi:10.1038/s41523-023-00597-0 · Cited by 20
Cross-Modal Retrieval and Semantic Refinement for Remote Sensing Image Captioning
Li, Z., Zhao, W., Du, X., Zhou, G., & Zhang, S. (2024) · Remote Sensing · doi:10.3390/rs16010196 · Cited by 20
Kim, Y., Jeong, H., Cho, J., & Shin, J. (2021) · Electronics · doi:10.3390/electronics10101170 · Cited by 17
Automatic medical report generation based on deep learning: A state of the art survey
Liu, X., Xin, J., Shen, Q., Huang, Z., & Wang, Z. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102486 · Cited by 22
Generating Image Captions using Deep Learning and Natural Language Processing
Gupta, S. C., Singh, N. R., Sharma, T., Tyagi, A., & Majumdar, R. (2021) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito51393.2021.9596486 · Cited by 16
An image caption model based on attention mechanism and deep reinforcement learning
Bai, T., Zhou, S., Pang, Y., Luo, J., Wang, H., & Du, Y. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1270850 · Cited by 18
“…processing, along with its potential value in terms of contributing to visually impaired individuals’ daily life assistance, graphic conversion, automatic title generation and machine intelligence (Hossain et al., 2019; Kang and Hu, 2022).”
BackgroundPhraseAug: An Augmented Medical Report Generation Model With Phrasebook
Mei, X., Yang, L., Gao, D., Cai, X., Han, J., & Liu, T. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3416190 · Cited by 18
Novel concept-based image captioning models using LSTM and multi-encoder transformer architecture
Osman, A. A. E., Shalaby, M. A. W., Soliman, M., & Elsayed, K. M. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-69664-1 · Cited by 19
Boosting image caption generation with feature fusion module
Xia, P., He, J., & Yin, J. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09110-2 · Cited by 16
Su, R. Z., Liu, J., Zhang, D., Cheng, C., & Ye, M. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.586197 · Cited by 16
“, 2018; Ristani and Tomasi, 2018), image captioning (Anderson et al., 2018; Hossain et al., 2019), and other fields (Hu et al.”
BackgroundTwo-Stage Merging Network for Describing Traffic Scenes in Intelligent Vehicle Driving System
Song, H., Zhu, J., & Jiang, Y. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3083656 · Cited by 16
“Then, the automatically described scenes need to properly combine natural language processing (NLP) and computer vision (CV), which are two important areas of artificial intelligence [7], [8], [9].”
BackgroundGenerating Image Captions based on Deep Learning and Natural language Processing
Sehgal, S., Sharma, J. N., & Chaudhary, N. (2020) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito48877.2020.9197977 · Cited by 16
“Filters[1]”
AltGen: AI-Driven Alt Text Generation for Enhancing EPUB Accessibility
Shen, Y., Zhang, H., Shen, Y., Wang, L., Shi, C., Du, S., & Tao, Y. (2025) · Proceedings of the 2025 International Conference on Artificial Intelligence and Computational Intelligence · doi:10.1145/3730436.3730449 · Cited by 18
“Semantically rich textual descriptions from visual inputs can be developed by means of transformer-based architectures s [11, 13, 22] which have shown strong performance in natural language processing and computer vision.”
GAVA: Spatial awareness in image captioning with geometric-aware visual attention
Hossain, M. A., Ye, Z., Hossen, M. B., Rahman, M. A., Islam, M. S., & Abdullah, M. I. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105435 · Cited by 16
Self-citationLearning Transferable Perturbations for Image Captioning
Wu, H., Liu, Y., Cai, H., & He, S. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3478024 · Cited by 15
Aesthetically Relevant Image Captioning
Zhong, Z., Zhou, F., & Qiu, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25485 · Cited by 17
Attention-based spatial-temporal graph transformer for traffic flow forecasting
Zhang, Q., Chang, W., Li, C., Yin, C., Su, Y., & Xiao, P. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08951-w · Cited by 18
“The proposal of the transformer model [16] leads to the application of attention mechanism gradually in various fields, such as natural language processing [31, 32], image classification [33, 34], and speech recognition [35].”
BackgroundSketch2Vis: Generating Data Visualizations from Hand-drawn Sketches with Deep Learning
Teng, Z., Fu, Q., White, J., & Schmidt, D. C. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00141 · Cited by 15
Reasoning over Vision and Language: Exploring the Benefits of Supplemental KnowledgePreprint
Shevchenko, V., Teney, D., Dick, A., & van den Hengel, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06013 · Cited by 22
“Vision-and-language tasks require joint processing of visual and textual data e.g. for image captioning (Anderson et al., 2018; Hossain et al., 2019) and VQA (Wu et al., 2017; Teney et al., 2017).”
Multi-GRU Based Automated Image Captioning for Smartphones
KESKİN, R., MORAL, Ö. T., Kılıç, V., & Onan, A. (2021) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu53274.2021.9477901 · Cited by 14
“Ancak, geleneksel RNN yapıları kaybolan ve patlayan gradyan problemlerine yol açtı˘gından, geli¸stirilen uzun kısa dönemli bellek (long short-term memory - LSTM) ve kapılı tekrarlayan hücre (gated recurrent unit - GRU) yapıları kullanılmaktadır [12].”
Describing UI Screenshots in Natural Language
Leiva, L. A., Hota, A., & Oulasvirta, A. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3564702 · Cited by 17
“A survey article by Hossain et al. [28] agglutinates recent research in this regard.”
BackgroundArtificial Intelligence-Based Diabetes Diagnosis with Belief Functions Theory
Ellouze, A., Kahouli, O., Ksantini, M., Alsaif, H., Aloui, A., & Kahouli, B. (2022) · Symmetry · doi:10.3390/sym14102197 · Cited by 14
Gharehbaghi, V., Farsangi, E. N., Yang, T. Y., Noori, M. N., & Kontoni, D. N. (2022) · Sensors · doi:10.3390/s22228986 · Cited by 14
“Several computer vision tasks, including image segmentation [13–15], object detection [16–18], image classification [11,19], and captioning [20,21], have been studied using pre-trained deep learning models.”
BackgroundAdversarial example detection using semantic graph matching
Gong, Y., Wang, S., Jiang, X., Yin, L., & Sun, F. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110317 · Cited by 14
Transformer model incorporating local graph semantic attention for image caption
Qian, K., Pan, Y., Xu, H., & Tian, L. (2023) · The Visual Computer · doi:10.1007/s00371-023-03180-7 · Cited by 14
Bi-SAN-CAP: Bi-Directional Self-Attention for Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2019) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta47822.2019.8946003 · Cited by 14
“[14] present a comprehensive survey of the topic.”
BackgroundSelf-citationTowards Mapping Images to Text Using Deep-Learning Architectures
Onita, D., Birlutiu, A., & Dinu, L. P. (2020) · Mathematics · doi:10.3390/math8091606 · Cited by 14
“deep-learning techniques are often used for image captioning tasks [1,2,4].”
BackgroundMethodLeveraging Deep Learning for Computer Vision: A Review
Alam, E., Sufian, A., Das, A. K., Bhattacharya, A., Ali, M. F., & Rahman, M. M. H. (2021) · International Arab Conference on Information Technology (ACIT) · doi:10.1109/acit53391.2021.9677361 · Cited by 16
Towards an AI Assistant for Power Grid Operators
Marot, A., Rozier, A., Dussartre, M., Crochepierre, L., & Donnot, B. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220191 · Cited by 28
“These advances enabled deeper and more practical large-scale real-time information processing, such as in computer vision [15], image understanding [16], nat-ural language processing [17], and recommendations [18].”
BackgroundDiagnosing fashion outfit compatibility with deep learning techniques
Balım, C., & Özkan, K. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119305 · Cited by 14
Reliable Extraction of Semantic Information and Rate of Innovation Estimation for Graph Signals
Kalfa, M., Yetim, S. Y., Atalık, A., Gok, M., Ge, Y., Li, R., Tong, W., Duman, T. M., & Arıkan, O. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221950 · Cited by 15
“Most popular semantic transformation techniques (especially in computer vision applications) include object detection [15], [16], [17], [18], [19], [20], semantic segmentation [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], and captioning [34], [35], [36], [37], [38], [39], [40], [41], [4…”
Unpaired Image Captioning by Image-Level Weakly-Supervised Visual Concept Recognition
Zhu, P., Wang, X., Luo, Y., Sun, Z., Zheng, W., Wang, Y., & Chen, C. W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3214090 · Cited by 17
“In the past few years, fully-supervised image captioning has been studied extensively [6].”
BackgroundFacilitated Deep Learning Models for Image Captioning
Azhar, I., Afyouni, I., & Elnagar, A. (2021) · Conference on Information Sciences and Systems · doi:10.1109/ciss50987.2021.9400209 · Cited by 14
“RNN is used in many sequence learning tasks such as machine translation, speech recognition and image captioning tasks [11].”
MethodAn Efficient Technique for Image Captioning Using Deep Neural Network
Phukan, B. B., & Panda, A. R. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-16-1056-1_38 · Cited by 13
Tang, C., Xie, L., Zhang, X., Hu, X., & Tian, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01465 · Cited by 15
“Typical examples include visual question answering [2, 6, 14, 46], image captioning [17, 43], visual grounding [18, 29, 32, 54], visual reasoning [51], etc .”
BackgroundYu, H., Cai, L., Min, H., & Su, X. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05825-9 · Cited by 14
“Despite of its preliminary success in medical practice, the basic FL convolutional neural networks (CNNs) [12, 13] still have some drawbacks, diminishing the overall system performance.”
Habib, M. A., Wadud, M. A. H., Patwary, M. F. K., Rahman, M. M., Mridha, M. F., Okuyama, Y., & Shin, J. (2024) · IEEE Access · doi:10.1109/access.2024.3435541 · Cited by 19
CLIP-Based Grid Features and Masking for Remote Sensing Image Captioning
Lin, Q., Wang, S., Ye, X., Wang, R., Yang, R., & Jiao, L. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3510414 · Cited by 16
Towards Fast and Robust Adversarial Training for Image Classification
Chen, E., & Lee, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69535-4_35 · Cited by 13
Alsayed, A., Arif, M., Qadah, T. M., & Alotaibi, S. S. (2023) · Applied Sciences · doi:10.3390/app131910894 · Cited by 15
Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning
Kumar, D., Srivastava, V., Popescu, D. E., & Hemanth, D. J. (2022) · Applied Sciences · doi:10.3390/app12136733 · Cited by 13
DLACEP: A Deep-Learning Based Framework for Approximate Complex Event Processing
Amir, A., Kolchinsky, I., & Schuster, A. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3526136 · Cited by 12
Exploring Data and Models in SAR Ship Image Captioning
Zhao, K., & Xiong, W. (2022) · IEEE Access · doi:10.1109/access.2022.3202193 · Cited by 14
“Image captioning [8], [9], [10] combines image processing and natural language processing, and it can generate descriptive sentences related to a given image.”
Background3DF-FCOS: Small object detection with 3D features based on FCOS
Yang, X., He, Y., Wu, J., Sun, W., Liu, T., & Ma, S. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103787 · Cited by 12
Noise-Aware Image Captioning with Progressively Exploring Mismatched Words
Fu, Z., Song, K., Zhou, L., & Yang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29097 · Cited by 15
“Image captioning is one of the important downstream tasks, which aims to automatically generate meaningful descriptions for images and has a wide range of applications in daily lives (Hossain et al. 2019; Baltrusaitis, Ahuja, and Morency 2019; Yang et al. 2023).”
BackgroundMulti-View Feature Fusion and Visual Prompt for Remote Sensing Image Captioning
Wang, S., Lin, Q., Ye, X., Liao, Y., Quan, D., Jin, Z., Hou, B., & Jiao, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3426359 · Cited by 13
Hateful Memes Classification using Machine Learning
Badour, J., & Brown, J. A. (2021) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci50451.2021.9659896 · Cited by 12
An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation
Moran, K., Yachnes, A., Purnell, G., Mahmud, J., Tufano, M., Cardenas, C. B., Poshyvanyk, D., & H'Doubler, Z. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00069 · Cited by 16
An intelligent approach to automatically discovering visual insights
Zhou, Y., Meng, X., Wu, Y., Wang, Y., Wang, Y., & Wu, Y. (2022) · Journal of Visualization · doi:10.1007/s12650-022-00894-z · Cited by 12
“Image captioning is a basic task in computer vision, aiming to generate a text summary of the semantic content of a natural image (Hossain et al. 2019; Mishra et al. 2022; Li et al. 2021).”
BackgroundA Survey on Attention-Based Models for Image Captioning
Osman, A. A. E., Shalaby, M. A. W., Soliman, M., & Elsayed, K. M. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140249 · Cited by 12
“Many surveys have been published for the deep learning techniques in image captioning [17, 38-42].”
BackgroundInteractive Interior Design Recommendation via Coarse-to-fine Multimodal Reinforcement Learning
Zhang, H., Sun, Y., Guo, W., Liu, Y., Lu, H., Lin, X. S., & Xiong, H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612420 · Cited by 13
“A higher similarity score indicates a higher chance of the object existing in the image.”
BackgroundImage Captioning using Facial Expression and Attention
Nezami, O. M., Dras, M., Wan, S., & Paris, C. L. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12025 · Cited by 13
A Sparse Transformer-Based Approach for Image Captioning
Zhou, L., Zhou, C., Chen, S., Huang, Y., & Liu, X. (2020) · IEEE Access · doi:10.1109/access.2020.3024639 · Cited by 12
“It does not only require detecting the objects and their relationships in an image but also needs to generate a syntactically and semantically correct sentence [1].”
BackgroundGeneration and Evaluation of Hindi Image Captions of Visual Genome
Singh, A., Meetei, L. S., Singh, T. D., & Bandyopadhyay, S. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-33-4084-8_7 · Cited by 12
En-De-Cap: An Encoder Decoder model for Image Captioning
Patwari, N., & Naik, D. (2021) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc51019.2021.9418414 · Cited by 12
“The general architecture of an image captioning network is a multi-model system (encoder-decoder) [1]wherein the CNN is used as encoder and RNN is used as decoder.”
MethodAutomatic image captioning system based on augmentation and ranking mechanism
Revathi, B. S., & Kowshalya, A. M. (2023) · Signal Image and Video Processing · doi:10.1007/s11760-023-02725-6 · Cited by 12
Image Captioning - A Deep Learning Approach Using CNN and LSTM Network
Voditel, P., Gurjar, A., Pandey, A., Jain, A., Sharma, N., & Dubey, N. (2023) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn58827.2023.00062 · Cited by 11
“The captioning process can be classified into categories like template-based image captioning, retrieval-based image captioning, and novel caption generation [1].”
BackgroundAttention-based multilayer GRU decoder for on-site glucose prediction on smartphone
Koca, Ö. A., Kabak, H. Ö., & Kılıç, V. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06424-y · Cited by 11
“In this context, the prediction of the glucose was performed using different PH, including 15, 30, 45, 60, 90, and 120.”
BVQA: Connecting Language and Vision Through Multimodal Attention for Open-Ended Question Answering
Bhuyan, M. S. M., Hossain, E., Sathi, K. A., Hossain, M. A., & Dewan, M. A. A. (2025) · IEEE Access · doi:10.1109/access.2025.3540388 · Cited by 11
Self-citationVisual Robustness Benchmark for Visual Question Answering (VQA)
Ishmam, M. F., Tashdeed, I., Saadat, T. A., Ashmafee, M. H., Kamal, A. R. M., & Hossain, M. A. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00645 · Cited by 21
“Our implementation of the PNP model utilized UnifiedQA [38] as the question-answering module and BLIP [41] for image-question matching and image captioning [29] modules.”
Self-citationDeep Gated Recurrent Unit for Smartphone-Based Image Captioning
Kılıç, V. (2021) · Sakarya University Journal of Computer and Information Sciences · doi:10.35377/saucis.04.02.866409 · Cited by 13
AESOP: Abstract Encoding of Stories, Objects, and Pictures
Ravi, H., Kafle, K., Cohen, S. D., Brandt, J., & Kapadia, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00206 · Cited by 13
Caption TLSTMs: combining transformer with LSTMs for image captioning
Yan, J., Xie, Y., Luan, X., Guo, Y., Gong, Q., & Feng, S. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-022-00228-7 · Cited by 12
A Visual Attention-Based Model for Bengali Image Captioning
Das, B., Pal, R., Majumder, M., Phadikar, S., & Sekh, A. A. (2023) · SN Computer Science · doi:10.1007/s42979-023-01671-x · Cited by 11
“Hence, these methods no longer expanded, and the neural network emerged in this field [12].”
BackgroundAttention-Based Spatial–Temporal Convolution Gated Recurrent Unit for Traffic Flow Forecasting
Zhang, Q., Chang, W., Yin, C., Xiao, P., Li, K., & Tan, M. (2023) · Entropy · doi:10.3390/e25060938 · Cited by 12
Sketch Captioning Using LSTM and BiLSTM
Dahikar, Y., Chikmurge, D. V., & Kharat, S. (2023) · International Conference on Networks, Multimedia and Information Technology (NMITCON) · doi:10.1109/nmitcon58196.2023.10275867 · Cited by 11
“These approaches heavily rely on recurrent neural networks (RNNs), often supported by long short-term memory (LSTM) units [2] and [3].”
BackgroundInfluentialSmith, M. J., Arora, N., Stone, C., Courteau, S., & Geach, J. E. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab424 · Cited by 14
Large scale datasets for Image and Video Captioning in Italian
Scaiella, A., Croce, D., & Basili, R. (2019) · Italian Journal of Computational Linguistics · doi:10.4000/ijcol.478 · Cited by 11
Image Captioning Methods and Metrics
Sargar, O., & Kinger, S. (2021) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci50559.2021.9396839 · Cited by 11
Transform Drug Discovery and Development With Generative Artificial Intelligence
Lavecchia, A. (2025) · Wiley eBooks · doi:10.1002/9781394280735.ch25 · Cited by 14
Amharic Language Image Captions Generation Using Hybridized Attention‐Based Deep Neural Networks
Solomon, R., & Abebe, M. (2023) · Applied Computational Intelligence and Soft Computing · doi:10.1155/2023/9397325 · Cited by 10
“However, this approach is rigid and cannot generate variable-length image descriptions [15, 18].”
Automated Quality Evaluation of Large-Scale Benchmark Datasets for Vision-Language Tasks
Zhao, R., Xie, Z., Zhuang, Y., & Yu, P. L. H. (2023) · International Journal of Neural Systems · doi:10.1142/s0129065724500096 · Cited by 11
Image Captioning Using VGG-16 Deep Learning Model
Jayaswal, V., Ji, S., Satyankar, Singh, V., Singh, Y., & Tiwari, V. (2024) · International Conference on Disruptive Technologies (ICDT) · doi:10.1109/icdt61202.2024.10489470 · Cited by 10
Learning Robust Scene Classification Model with Data Augmentation Based on Xception
Chen, H., Yang, Y., & Zhang, S. (2020) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1575/1/012009 · Cited by 10
A Deep-Learning-Based Method for Diagnosing Time-Varying Plasma Adopting Microwaves
Liu, H., Yang, M., Liu, Y., Geng, J., & Tang, J. (2021) · IEEE Transactions on Plasma Science · doi:10.1109/tps.2021.3065343 · Cited by 10
“During the past few years, deep learning (DL) [14] has been applied in various fields, such as computer vision [15], natural language processing [16], and communication system [17].”
BackgroundUppal, S., Madan, A., Bhagat, S., Yu, Y., & Shah, R. R. (2021) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3444685.3446302 · Cited by 10
“In general, structured latent spaces has aided downstream task performance in diverse fields such as image captioning [8] and language inferences [15, 27].”
BackgroundLeveraging CLIP for Image Emotion Recognition.
Bondielli, A., & Passaro, L. (2021) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 10
Challenges and opportunities of image and video retrieval
Qiu, G. (2022) · Frontiers in Imaging · doi:10.3389/fimag.2022.951934 · Cited by 10
“Recent years has seen much progress in image and video tagging and captioning (Hossain et al., 2019) where words and sentences are automatically generated to describe the visual content.”
Background
Metadata
- Authors
- Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H.
- Affiliations
- Murdoch University (AU)
- Venue
- ACM Computing Surveys
- Published
- 2019-02-04
- Volume
- 51
- Issue
- 6
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2896348597
- Cited by
- 879
- References
- 166
- Citations by year
- 2026: 55 · 2025: 113 · 2024: 124 · 2023: 167 · 2022: 153 · 2021: 127 · 2020: 104 · 2019: 36
- FWCI
- 47.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Murdoch University Research Portal (submittedVersion)
- Funders
- Australian Research Council (DE120102960)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- image captioning, object recognition, sentence generation, evaluation metrics, deep learning architectures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
