The DOI matches its Crossref record; 167 of the 167 references it cites were found in a registry or an index.
A Comprehensive Survey of Deep Learning for Image Captioning
Abstract
Generating a description of an image is called image captioning. Image captioning requires recognizing the important objects, their attributes, and their relationships in an image. It also needs to generate syntactically and semantically correct sentences. Deep-learning-based techniques are capable of handling the complexities and challenges of image captioning. In this survey article, we aim to present a comprehensive review of existing deep-learning-based image captioning techniques. We discuss the foundation of the techniques to analyze their performances, strengths, and limitations. We also discuss the datasets and the evaluation metrics popularly used in deep-learning-based automatic image captioning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most prior works have relied on a set of metrics inspired by similar setups in natural language generation, where radiology report text is treated as generic text.22”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [22]
In the citing paper
“If we take a broader view, many works are using cross-modal deep generative models (such as GAN) in data synthesis, such as from video to audio [267], from text to image and vice versa [268,269].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [268]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“The architecture of this method has ResNet [53] and two LSTMs called Skel-LSTM and Attr-LSTM.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“Moreover, recent object detection and classification methods [73, 127] show that deep, hierarchical methods are better at learning than shallower ones.”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5×
“Kiros et al. [69] proposed a method that is built on a Log-Bilinear model and used AlexNet to extract visual features.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM [54] networks are a type of RNN that have special units in addition to standard units.”
BackgroundMethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 4×
“The bottom up attention mechanism uses Faster R-CNN [116] for region proposals that can select salient regions of an image .”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“For example, Convolutional Neural Networks (CNN) [79] are widely used for feature learning, and a classifier such as Softmax is used for classification.”
MethodDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 1×
“d techniques and (2) Deep machine learning based techniques. In traditional machine learning, hand crafted features such as Local Binary Patterns (LBP) [107], Scale-Invariant Feature Transform (SIFT) [87], the Histogram of Oriented Gradients (HOG) [27], and a combination of such features are widely used. In these tech…”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Flagged first, then by citations
Research works printed but not linked: 58
- 10.5555/1626394.1626406 · DOI
- Aker Ahmet, 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 1250--1258
- Bahdanau Dzmitry, 2015, International Conference on Learning Representations (ICLR’15)
- Cristian Bodnar. 2018. Text to image synthesis using generative adversarial networks. arXiv preprint arXiv:1805.00676. Cristian Bodnar. 2018. Text to image synthesis using generative adversarial networks. arXiv preprint arXiv:1805.00676. · arXiv
- Chen Xinlei, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2422--2431
- Cho Kyunghyun, 2014, Dzmitry Bahdanau, and Yoshua Bengio.
- Denoual Etienne, Companion
- Fang Hao, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1473--1482
- 10.5555/1888089.1888092 · DOI
- Fedus William, 2018, Dai
- FitzGerald Nicholas, 2013, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 1914--1925
- Frome Andrea, 2013, Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems. 2121--2129.
- Gehring Jonas, 2016, Dauphin
- Gehring Jonas, 2017, Dauphin
- 10.5555/1870658.1870698 · DOI
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680.
- Gregor Karol, 2015, Proceedings of Machine Learning Research. 1462--1471
- Grubinger Michael, 2006, International Workshop Ontoimage, 5
- Isola Phillip, Proceedings of the IEEE International Conference on Computer Vision (CVPR’17)
- Max Jaderberg Karen Simonyan Andrew Zisserman and Koray Kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025. Max Jaderberg Karen Simonyan Andrew Zisserman and Koray Kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025.
- Jang Eric, 2017, International Conference on Learning Representations (ICLR’17)
- Kazemzadeh Sahar, 2014, Berg
- Kiros Ryan, 2014, Proceedings of the 31st International Conference on Machine Learning (ICML’14)
- Kiros Ryan, Workshop on Neural Information Processing Systems (NIPS’14)
- Konda Vijay R., 2000, Tsitsiklis
- Krizhevsky Alex, 2012, Hinton
- 10.5555/2390524.2390575 · DOI
- Kuznetsova Polina, 2014, TREETALK: Composition and compression of trees for image descriptions.TACL 2, 10
- Lebret RÃl’mi, 2015, Workshop on International Conference on Learning Representations (ICLR’15)
- Li Siming, 2011, Proceedings of the 15th Conference on Computational Natural Language Learning. Association for Computational Linguistics, 220--228
- Lin Chin-Yew, 2004, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 8
- Liu Chenxi, 2017, Yuille
- Maddison Chris J., 2017, International Conference on Learning Representations (ICLR’17)
- Mao Junhua, 2015, International Conference on Learning Representations (ICLR’15)
- Mao Junhua, 2014, Yuille
- Mitchell Margaret, 2012, Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 747--756
- 10.5555/1873738.1873752 · DOI
- Mitchell Margaret, 2013, Kees Van Deemter, and Ehud Reiter
- 10.1145/1273496.1273577 · DOI
- 10.5555/645313.649410 · DOI
- Ordonez Vicente, 2011, Berg
- Ranzato Marc’Aurelio, 2016, International Conference on Learning Representations (ICLR’16)
- Zhou Ren Hailin Jin Zhe Lin Chen Fang and Alan Yuille. 2015. Multi-instance visual-semantic embedding. arXiv preprint arXiv:1512.06963. Zhou Ren Hailin Jin Zhe Lin Chen Fang and Alan Yuille. 2015. Multi-instance visual-semantic embedding. arXiv preprint arXiv:1512.06963. · arXiv
- Vadivel Karthikeyan Shanmuga, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3241--3250
- Simonyan Karen, 2015, International Conference on Learning Representations (ICLR’15)
- Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203. Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203. · arXiv
- Sutskever Ilya, 2014, Le
- van Deemter Kees, Proceedings of the 4th International Natural Language Generation Conference. Association for Computational Linguistics, 130--132
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998--6008.
- 10.5555/1708322.1708334 · DOI
- Wang Qingzhong, 2018, Chan
- Wang Yufei, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17)
- Yang Zhilin, 30th Conference on Neural Image Processing System (NIPS’16)
- Xu Kelvin, 2015, International Conference on Machine Learning. 2048--2057
- Yang Linjie, 2016, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16)
- Yu Lantao, 2017, Proceedings of the 31st AAAI Conference on Artificial Intelligence.
- Matthew, 2014, Zeiler and Rob Fergus
- Zhang Li, 2017, Hospedales
Cited by
Sorted by citations
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
“The use of deep learning for image captioning has been surveyed very recently [230], [231], and tasks that generate text given textual inputs are discussed in Sections IV-E–IV-G.”
BackgroundClimbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082
“One approach to providing grounding is to train distributional models on corpora augmented with perceptual data, such as photos (Hossain et al., 2019) or other modalities (Kiela and Clark, 2015; Kiela et al., 2015).”
BackgroundConvolutional neural networks in medical image understanding: a survey
Sarvamangala, D. R., & Kulkarni, R. V. (2021) · Evolutionary Intelligence · doi:10.1007/s12065-020-00540-3 · Cited by 1,043
“Researchers are also working on image captioning (textual representations of the image) [29].”
BackgroundZhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“video [179] while text-to-image generation aims to generate a realistic image that matches the given text description [180].”
BackgroundLearning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting
Guo, S., Lin, Y., Wan, H., Li, X., & Cong, G. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3056502 · Cited by 800
“It has been widely used and achieved great successes in various tasks such as natural language processing (NLP) [39], [40], image caption [41] and speech recognition [42].”
BackgroundDeep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625
“If we take a broader view, many works are using cross-modal deep generative models (such as GAN) in data synthesis, such as from video to audio [266], from text to image and vice versa [267, 268].”
BackgroundA novel genetic LSTM model for wind power forecast
Shahid, F., Zameer, A., & Muneeb, M. (2021) · Energy · doi:10.1016/j.energy.2021.120069 · Cited by 543
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 580
A review of deep learning with special emphasis on architectures, applications and recent trends
Sengupta, S., Basak, S., Saikia, P., Paul, S., Tsalavoutis, V. A., Atiah, F. D., Ravi, V., & Peters, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105596 · Cited by 490
Thin MobileNet: An Enhanced MobileNet Architecture
Sinha, D., & El‐Sharkawy, M. (2019) · IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon47517.2019.8993089 · Cited by 436
“In real-time applications such as image classification [8], captioning [9], object detection [10,23,33] and semantic segmentation [34] in autonomous driving, the inference time and accuracy are very important factors to one's safety.”
BackgroundCamacho, D., Panizo-LLedot, Á., Bello-Orgaz, G., González-Pardo, A., & Cambria, E. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.05.009 · Cited by 359
Visual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 364
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373
“Compared to previous surveys on image captioning [16], [17], [18], [21], [22], we provide a comprehensive and updated view on deep learning-based generative captioning models.”
BackgroundFoundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304
“StandardCaptioning( Cap )loss[45] also aims to predict the next token given previous tokens and images.”
Computer vision applications in construction safety assurance
Fang, W., Ding, L., Love, P. E. D., Luo, H., Li, H., Pena-Mora, F. A., Zhong, B., & Zhou, C. (2019) · Automation in Construction · doi:10.1016/j.autcon.2019.103013 · Cited by 328
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286
Vision Transformers in medical computer vision—A contemplative retrospection
Parvaiz, A., Khalid, M. A., Zafar, R., Ameer, H., Ali, M. U., & Fraz, M. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106126 · Cited by 302
Relation between prognostics predictor evaluation metrics and local interpretability SHAP values
Baptista, M. L., Goebel, K., & Henriques, E. M. P. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103667 · Cited by 283
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257
“Beyond text-to-text generation tasks, recent years have witnessed a growing interest in visuallanguage (VL) generation tasks, such as describing visual scenes [46], and answering visual-related questions [81].”
BackgroundMedical deep learning—A systematic meta-review
Egger, J., Gsaxner, C., Pepe, A., Pomykala, K. L., Jonske, F., Kurz, M., Li, J., & Kleesiek, J. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106874 · Cited by 289
“Object detection [75–77] Image segmentation [ 78 , 79 ] Face recognition [80–82] Action/motion recognition [ 83 , 84 ] Biometric recognition [ 85 , 86 ] Image super-resolution [87] Image captioning [88] Data augmentation [89] Generative adversarial networks [90]”
Improved Few-Shot Visual Classification
Bateni, P., Goyal, R., Masrani, V., Wood, F., & Sigal, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01450 · Cited by 248
Machine Learning for Clinical Outcome Prediction
Shamout, F. E., Zhu, T., & Clifton, D. A. (2020) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3007816 · Cited by 257
Methods for Pruning Deep Neural Networks
Vadera, S., & Ameen, S. (2022) · IEEE Access · doi:10.1109/access.2022.3182659 · Cited by 250
“There are many different types of deep learning networks, including convolutional neural networks (CNNs), recurrent neural networks and recursive neural networks [75,139].”
BackgroundDeep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction
Nasser, M., & Yusof, U. K. (2023) · Diagnostics · doi:10.3390/diagnostics13010161 · Cited by 249
“To mitigate the issue of the vanishing and exploding gradients of RNN, new models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) were introduced [36].”
MethodInfluentialArtificial intelligence in drug discovery: applications and techniques
Deng, J., Yang, Z., Ojima, I., Samaras, D., & Wang, F. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab430 · Cited by 244
Imran, M., Ofli, F., Caragea, D., & Torralba, A. B. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102261 · Cited by 241
Adversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 228
“T2I can be seen as the inverse of image captioning [15], where the input is an image and the output is a textual description of that image.”
BackgroundDynamic Graph Enhanced Contrastive Learning for Chest X-Ray Report Generation
Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., & Chang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00325 · Cited by 206
Designing ECG monitoring healthcare system with federated transfer learning and explainable AI
Raza, A., Tran, K. P., Koehl, L., & Li, S. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107763 · Cited by 205
“However, thanks to the internet, IoT devices and the increasing computational power, nowadays we can see deep learning revolutionizing nearly every field, including healthcare [5], economics [6], manufacturing [7], agriculture [8], and military [9].”
BackgroundEvaluating progress in automatic chest X-ray radiology report generation
Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E. P., Fonseca, E. K. U. N., Lee, H. M. H., Abad, Z. S. H., Ng, A. Y., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100802 · Cited by 200
“Most prior works have relied on a set of automated metrics inspired by similar setups in natural language generation, where radiology report text is treated as generic text 21 .”
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 219
“However, instead of a pairwise comparison like in [58], we have organized our paper to feature a separate section for each method category and follow the same order of category in our discussion (section 5).”
BackgroundMethodScalable Deep Learning on Distributed Infrastructures
Mayer, R., & Jacobsen, H. (2020) · ACM Computing Surveys · doi:10.1145/3363554 · Cited by 232
“, automatically colorize grayscale images) [75], image captioning [68] (i.”
BackgroundThe future of coastal monitoring through satellite remote sensing
Vitousek, S., Buscombe, D., Vos, K., Barnard, P. L., Ritchie, A. C., & Warrick, J. A. (2022) · Cambridge Prisms Coastal Futures · doi:10.1017/cft.2022.4 · Cited by 192
Deep Learning (CNN) and Transfer Learning: A Review
Gupta, J., Pathak, S. K., & Kumar, G. (2022) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2273/1/012029 · Cited by 224
“CNN's coverage of DL has made it a household name [10], [11].”
BackgroundCaptioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182
Translation between Molecules and Natural Language
Edwards, C., Lai, T., Ros, K., Honke, G., Cho, K., & Ji, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.26 · Cited by 324
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01363 · Cited by 166
“These models have demonstrated profound capabilities in various applications and significantly enhance the performance in image reasoning tasks [5, 18, 20, 30, 31, 36, 38, 42, 47].”
BackgroundDescription Generation for Remote Sensing Images Using Attribute Attention Mechanism
Zhang, X., Wang, X., Tang, X., Zhou, H., & Li, C. (2019) · Remote Sensing · doi:10.3390/rs11060612 · Cited by 153
“There are three mai categories of the existing image captioning m thods [28]: template, retrieval, and encoder-decoderbased im ge capti ni g.”
BackgroundScreen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K. I., Yu, L., Shan, Q., Nichols, J. W., Wu, J., Fleizach, C., et al. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 145
“To describe content in pictures, prior work used deep learning models to generate natural language descriptions of images [44, 46], and some accessibility improvement research has also leveraged crowdsourcing to generate image captions [35, 39, 40].”
MethodComprehensive Image Captioning via Scene Graph Decomposition
Zhong, Y., Wang, L., Chen, J., Yu, D., & Li, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_13 · Cited by 140
Video Processing Using Deep Learning Techniques: A Systematic Literature Review
Sharma, V., Gupta, M., Kumar, A., & Mishra, D. (2021) · IEEE Access · doi:10.1109/access.2021.3118541 · Cited by 153
“Another motivation is that in the past ten years, so many review papers are present for deep learning for image processing[17][18]; similarly, deep learning techniques for video processing are needed.”
BackgroundPLA: Language-Driven Open-Vocabulary 3D Scene Understanding
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., & Qi, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00677 · Cited by 131
“As image captioning is a fundamental task in VL research area [18], various foundation models [40, 1, 30] trained with massive samples are readily available for solving this task.”
BackgroundMeta Graph Transformer: A Novel Framework for Spatial–Temporal Traffic Prediction
Ye, X., Fang, S., Sun, F., Zhang, C., & Xiang, S. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.12.033 · Cited by 128
Levels of explainable artificial intelligence for human-aligned conversational explanations
Dazeley, R., Vamplew, P. W., Foale, C., Young, C., Aryal, S., & Cruz, F. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103525 · Cited by 138
“For example, Deep Reinforcement Learning [116–119] is a class of algorithms that essentially utilise a combination of two distinct algorithms Reinforcement Learning and Deep Neural Networks.”
BackgroundPhysical Adversarial Attack Meets Computer Vision: A Decade Survey
Wei, H., Tang, H., Jia, X., Wang, Z., Yu, H., Li, Z., Satoh, S., Gool, L. V., & Wang, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3430860 · Cited by 152
“7 for illustrations.”
Jiang, X., & Xu, C. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11195772 · Cited by 128
SD-RSIC: Summarization-Driven Deep Remote Sensing Image Captioning
Sümbül, G., Nayak, S., & Demir, B. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3031111 · Cited by 119
A Systematic Review of Urban Navigation Systems for Visually Impaired People
Eltaher, F., Taha, A., Courtney, J., & McKeever, S. (2021) · Sensors · doi:10.3390/s21093103 · Cited by 110
“While there are different datasets available for this task [129], none of them were captured from the BVIP perspective.”
BackgroundVisual to Text: Survey of Image and Video Captioning
Li, S., Tao, Z., Li, K., & Fu, Y. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2892755 · Cited by 112
Improved Automatic Summarization of Subroutines via Attention to File Context
Haque, S., LeClair, A., Wu, L., & McMillan, C. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387449 · Cited by 105
“) in the same input sequence, while in other research areas such as image captioning [20] it has been shown that better results are achieved when different input types are processed separately.”
BackgroundAutomatic Image Captioning Based on ResNet50 and LSTM with Soft Attention
Chu, Y., Xiao, Y., Yu, L., Sergei, M., & Wang, Z. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8909458 · Cited by 108
“Much research effort has been devoted to automatic image captioning, and it can be categorized into templatebased image captioning, retrieval-based image captioning, and novel image caption generation [5].”
BackgroundAlpha-CLIP: A CLIP Model Focusing on Wherever you Want
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01237 · Cited by 151
“While CLIP captures the content of the entire image, it is also crucial to focus on the regions of interest to enable a finer understanding [14, 18, 21, 37, 48, 72] and controllable content generation [22, 33, 46, 55].”
Automatic captioning for medical imaging (MIC): a rapid review of literature
Beddiar, D. R., Oussalah, M., & Seppänen, T. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10270-w · Cited by 103
“This is due, in fact, to both the inherent properties of medical images with their complex anatomy structures and to the fact that the annotations should only focus on clinically important aspects that are relevant to the diagnosis, rather than mapping all objects in the image as in a generic image captioning system (…”
BackgroundZhang, Y., Liu, P., & Biljecki, F. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.03.008 · Cited by 107
“Machine reasoning is used to deconstruct and understand the content of the image, then generate a natural semantic description of the given scene (Hossain et al., 2019).”
Large Language Models versus Natural Language Understanding and Generation
Karanikolas, N., Manga, E., Samaridi, N., Tousidou, E., & Vassilakopoulos, M. (2023) · Proceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics · doi:10.1145/3635059.3635104 · Cited by 102
Machine Learning-Based Intrusion Detection Methods in IoT Systems: A Comprehensive Review
Kikissagbe, B. R., & Adda, M. (2024) · Electronics · doi:10.3390/electronics13183601 · Cited by 142
“These results confirm the potential of SVM as a robust tool for anomaly detection in IoT network security [32].”
ResultIshmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 107
“A similar mechanism is employed in VQA following the success of attention in standard computer vision tasks like object recognition [228] and multimodal tasks like image captioning [229, 8].”
BackgroundMethodInfluentialFully automated carbonate petrography using deep convolutional neural networks
Koeshidayatullah, A., Morsilli, M., Lehrmann, D. J., Al-Ramadan, K. A., & Payne, J. L. (2020) · Marine and Petroleum Geology · doi:10.1016/j.marpetgeo.2020.104687 · Cited by 100
Orderless Recurrent Models for Multi-Label Classification
Yazici, V. O., González-García, A., Arnau Ramisa, A., Twardowski, B., & van de Weijer, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01345 · Cited by 98
Domingos, L. C. F., Santos, P. E., Skelton, P. S. M., Brinkworth, R. S. A., & Sammut, K. (2022) · Sensors · doi:10.3390/s22062181 · Cited by 93
“There have been significant recent developments in Deep Learning (DL) methods, mostly pushed by object detection and classification in images [45,46] and visual question answering [47].”
BackgroundxLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299
Image captioning with transformer and knowledge graph
Zhang, Y., Shi, X., Mi, S., & Yang, X. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.12.020 · Cited by 87
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 85
Incorporating attentive multi-scale context information for image captioning
Prudviraj, J., Yenduri, S., & Mohan, C. K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11895-9 · Cited by 85
“Several pioneering works [20, 23, 39, 55, 59, 64, 65] rely on such combination of CNN+RNN encoder-decoder approach.”
BackgroundMethodMultimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 87
“education, digital libraries, and web searching [95].”
A survey on automatic generation of medical imaging reports based on deep learning
Pang, T., Li, P., & Zhao, L. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01113-y · Cited by 88
“The automatic generation of diagnostic reports is inspired by image captioning [2], which combines computer vison (CV) and natural language processing (NLP) to provide a comprehensive understanding of medical images.”
MethodDeep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 79
Towards goal-oriented semantic signal processing: Applications and future challenges
Kalfa, M., Gok, M., Atalık, A., Tegin, B., Duman, T. M., & Arıkan, O. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2021.103134 · Cited by 81
“A comprehensive survey of image and video captioning models can be found in [110] and [103], respectively.”
BackgroundJoint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation
Yang, Y. Q., Yu, J., Zhang, J., Han, W., Jiang, H., & Huang, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3122542 · Cited by 78
“The earliest works generate image captions with template-based or retrieval-based methods [11].”
BackgroundRyu, J., Kim, Y.-H., Kim, T., & Jung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-32514-7 · Cited by 81
Das, D. K. (2025) · Urban Science · doi:10.3390/urbansci9030072 · Cited by 91
“For example, DL applications such as image and video recognition are essential for urban security and surveillance, helping to identify objects, faces, and scenes in real time [119].”
Enhancing Few-Shot Image Classification with Unlabelled Examples
Bateni, P., Barber, J., van de Meent, J.-W., & Wood, F. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00166 · Cited by 69
A Systematic Literature Review on Image Captioning
Staniūtė, R., & Šešok, D. (2019) · Applied Sciences · doi:10.3390/app9102024 · Cited by 68
“There are only a few SLRs that have been conducted for image captioning until now [6–9], though with such fast progress and increasing popularity in this field we find it necessary that they continue to be undertaken.”
BackgroundMethodKnowledge Graphs for Multi-modal Learning: Survey and Perspective
Chen, Z., Zhang, Y., Fang, Y., Geng, Y., Guo, L., Chen, J., Liu, X., Pan, J. Z., Zhang, N., Chen, H., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103124 · Cited by 79
Text to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 78
“[18] present a comprehensive survey of the topic.”
BackgroundSelf-citationReale-Nosei, G., Amador-Domínguez, E., & Serrano, E. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103264 · Cited by 72
A Multi-Level Attention Model for Remote Sensing Image Captions
Li, Y., Fang, S., Jiao, L., Liu, R., & Shang, R. (2020) · Remote Sensing · doi:10.3390/rs12060939 · Cited by 72
Neural attention for image captioning: review of outstanding methods
Zohourianshahzadi, Z., & Kalita, J. K. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10092-2 · Cited by 68
“Recently several high-quality surveys on image captioning with deep neural networks (Hossain et al. 2019; Liu et al. 2019b) and other techniques such as template generation and slot filling (Bai and An 2018; Sharma et al.”
BackgroundVaBUS: Edge-Cloud Real-Time Video Analytics via Background Understanding and Subtraction
Wang, H., Li, Q., Sun, H., Chen, Z., Hao, Y., Peng, J., Yuan, Z., Fu, J., & Jiang, Y. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221995 · Cited by 61
“…of deep learning models have been proposed for various vision applications, including image classification [51], instance segmentation [52], object detection [53], face recognition [54], image captioning [55], etc. Recent works show that it is inefficient to perform inference for every video frame.”
BackgroundA Joint-Training Two-Stage Method For Remote Sensing Image Captioning
Ye, X., Wang, S., Gu, Y., Wang, J., Wang, R., Hou, B., Giunchiglia, F., & Jiao, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3224244 · Cited by 71
“Therefore, RSI captioning has attracted increasing attention from researchers [1], [2], [3], [4].”
BackgroundExplainable reinforcement learning for broad-XAI: a conceptual framework and survey
Dazeley, R., Vamplew, P. W., & Cruz, F. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08423-1 · Cited by 86
“Hence, function approximation techniques such as Neural Networks are frequently utilised, giving rise to the field of Deep RL (DRL) [163, 144, 15, 66, 109, 226].”
MethodDynamic Attention Guided Multi-Trajectory Analysis for Single Object Tracking
Wang, X., Chen, Z., Tang, J., Luo, B., Wang, Y., Tian, Y., & Wu, F. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3056684 · Cited by 57
Zhang, J., Xiang, R., Kuang, Z., Wang, B., & Li, Y. (2024) · Heritage Science · doi:10.1186/s40494-024-01334-x · Cited by 60
“For the task planning aspect, we will define necessary task steps based on prior knowledge (e.g., when an image is inputted, the Image Caption model [18] must be called to supplement LLM’s understanding of the image), to provide essential inputs for the final response (refer to "Task parsing"…”
MethodA trend graph attention network for traffic prediction
Wang, C., Tian, R., Hu, J., & Ma, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.12.048 · Cited by 54
Deep learning in fringe projection: A review
Liu, H., Yan, N., Shao, B., Yuan, S., & Zhang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127493 · Cited by 58
Prompt-Based Learning for Unpaired Image Captioning
Zhu, P., Wang, X., Zhu, L., Sun, Z., Zheng, W., Wang, Y., & Chen, C. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3265842 · Cited by 57
Sharma, H., & Padha, D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10488-2 · Cited by 51
Kim, K., Cho, K., Jang, R., Kyung, S., Lee, S., Ham, S., Choi, E., Hong, G.-S., & Kim, N. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0818 · Cited by 54
“AI models create relevant and kjronline.org descriptive captions for the visual inputs provided in visual captioning [49].”
Spatial dynamic graph convolutional network for traffic flow forecasting
Li, H., Yang, S., Song, Y., Luo, Y., Li, J., & Zhou, T. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04271-z · Cited by 51
“Attention mechanism has been widely applied in natural language processing (NLP) [32, 33], image recognition [34, 35], protein identification [36, 37], recommended system [38] and etc. Self-attention which emphasizes many-to-many is a variant of Attention mechanism.”
BackgroundTsai, W.-L., Le, P.-L., Ho, W.-F., Chi, N.-W., Lin, J. J., Tang, S., & Hsieh, S. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105863 · Cited by 50
Image captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49
da Costa, P. R. D. O., Akçay, A., Zhang, Y., & Kaymak, U. (2023) · International Journal of Prognostics and Health Management · doi:10.36001/ijphm.2019.v10i4.2623 · Cited by 51
“CNNs are notable for being able to extract spatial information from 2D and 3D data (Hossain et al., 2019).”
BackgroundMitigating Gender Bias in Captioning Systems
Tang, R., Du, M., Li, Y., Liu, Z., Zou, N., & Hu, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449950 · Cited by 50
Image Captioning Using Inception V3 Transfer Learning Model
Degadwala, S., Vyas, D., Biswas, H., Chakraborty, U., & Saha, S. (2021) · International Conference on Communication and Electronics Systems (ICCES) · doi:10.1109/icces51350.2021.9489111 · Cited by 48
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 146
“We present details for only eight tasks by excluding visual description generation (Aafaq et al., 2018; Hossain et al., 2019) and question answering (Wu et al.”
BackgroundMethodInfluentialRetrieval-augmented Image Captioning
Ramos, R., Elliott, D., & Martins, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.266 · Cited by 61
A Study of ConvNeXt Architectures for Enhanced Image Captioning
Ramos, L., Casas, E., Romero, C., Rivas, F., & Morocho-Cayamcela, M. E. (2024) · IEEE Access · doi:10.1109/access.2024.3356551 · Cited by 47
High-Throughput Vector Similarity Search in Knowledge Graphs
Mohoney, J., Pacaci, A., Chowdhury, S. R., Mousavi, A., Ilyas, I. F., Minhas, U. F., Pound, J., & Rekatsinas, T. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3589777 · Cited by 48
“Modern machine learning models are increasingly used to embed complex data such as images, text, or entities in knowledge graphs into vector representations that retain semantically meaningful information [5, 6, 9, 18, 29].”
BackgroundAutomatic image captioning combining natural language processing and deep neural networks
Rinaldi, A. M., Russo, C., & Tommasino, C. (2023) · Results in Engineering · doi:10.1016/j.rineng.2023.101107 · Cited by 45
Boosting convolutional image captioning with semantic content and visual relationship
Bai, C., Zheng, A., Huang, Y., Pan, X., & Chen, N. (2021) · Displays · doi:10.1016/j.displa.2021.102069 · Cited by 49
Metadata
- Authors
- Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H.
- Affiliations
- Murdoch University (AU)
- Venue
- ACM Computing Surveys
- Published
- 2019-02-04
- Volume
- 51
- Issue
- 6
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2896348597
- Cited by
- 879
- References
- 166
- Citations by year
- 2026: 55 · 2025: 113 · 2024: 124 · 2023: 167 · 2022: 153 · 2021: 127 · 2020: 104 · 2019: 36
- FWCI
- 47.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Murdoch University Research Portal (submittedVersion)
- Funders
- Australian Research Council (DE120102960)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- image captioning, object recognition, sentence generation, evaluation metrics, deep learning architectures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
