The DOI matches its Crossref record; 167 of the 167 references it cites were found in a registry or an index.
A Comprehensive Survey of Deep Learning for Image Captioning
Abstract
Generating a description of an image is called image captioning. Image captioning requires recognizing the important objects, their attributes, and their relationships in an image. It also needs to generate syntactically and semantically correct sentences. Deep-learning-based techniques are capable of handling the complexities and challenges of image captioning. In this survey article, we aim to present a comprehensive review of existing deep-learning-based image captioning techniques. We discuss the foundation of the techniques to analyze their performances, strengths, and limitations. We also discuss the datasets and the evaluation metrics popularly used in deep-learning-based automatic image captioning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most prior works have relied on a set of metrics inspired by similar setups in natural language generation, where radiology report text is treated as generic text.22”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [22]
In the citing paper
“If we take a broader view, many works are using cross-modal deep generative models (such as GAN) in data synthesis, such as from video to audio [267], from text to image and vice versa [268,269].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [268]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“The architecture of this method has ResNet [53] and two LSTMs called Skel-LSTM and Attr-LSTM.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“Moreover, recent object detection and classification methods [73, 127] show that deep, hierarchical methods are better at learning than shallower ones.”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 5×
“Kiros et al. [69] proposed a method that is built on a Log-Bilinear model and used AlexNet to extract visual features.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM [54] networks are a type of RNN that have special units in addition to standard units.”
BackgroundMethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 4×
“The bottom up attention mechanism uses Faster R-CNN [116] for region proposals that can select salient regions of an image .”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“For example, Convolutional Neural Networks (CNN) [79] are widely used for feature learning, and a classifier such as Softmax is used for classification.”
MethodDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · cited 1×
“d techniques and (2) Deep machine learning based techniques. In traditional machine learning, hand crafted features such as Local Binary Patterns (LBP) [107], Scale-Invariant Feature Transform (SIFT) [87], the Histogram of Oriented Gradients (HOG) [27], and a combination of such features are widely used. In these tech…”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Flagged first, then by citations
Research works printed but not linked: 58
- 10.5555/1626394.1626406 · DOI
- Aker Ahmet, 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 1250--1258
- Bahdanau Dzmitry, 2015, International Conference on Learning Representations (ICLR’15)
- Cristian Bodnar. 2018. Text to image synthesis using generative adversarial networks. arXiv preprint arXiv:1805.00676. Cristian Bodnar. 2018. Text to image synthesis using generative adversarial networks. arXiv preprint arXiv:1805.00676. · arXiv
- Chen Xinlei, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2422--2431
- Cho Kyunghyun, 2014, Dzmitry Bahdanau, and Yoshua Bengio.
- Denoual Etienne, Companion
- Fang Hao, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1473--1482
- 10.5555/1888089.1888092 · DOI
- Fedus William, 2018, Dai
- FitzGerald Nicholas, 2013, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 1914--1925
- Frome Andrea, 2013, Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems. 2121--2129.
- Gehring Jonas, 2016, Dauphin
- Gehring Jonas, 2017, Dauphin
- 10.5555/1870658.1870698 · DOI
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680.
- Gregor Karol, 2015, Proceedings of Machine Learning Research. 1462--1471
- Grubinger Michael, 2006, International Workshop Ontoimage, 5
- Isola Phillip, Proceedings of the IEEE International Conference on Computer Vision (CVPR’17)
- Max Jaderberg Karen Simonyan Andrew Zisserman and Koray Kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025. Max Jaderberg Karen Simonyan Andrew Zisserman and Koray Kavukcuoglu. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025.
- Jang Eric, 2017, International Conference on Learning Representations (ICLR’17)
- Kazemzadeh Sahar, 2014, Berg
- Kiros Ryan, 2014, Proceedings of the 31st International Conference on Machine Learning (ICML’14)
- Kiros Ryan, Workshop on Neural Information Processing Systems (NIPS’14)
- Konda Vijay R., 2000, Tsitsiklis
- Krizhevsky Alex, 2012, Hinton
- 10.5555/2390524.2390575 · DOI
- Kuznetsova Polina, 2014, TREETALK: Composition and compression of trees for image descriptions.TACL 2, 10
- Lebret RÃl’mi, 2015, Workshop on International Conference on Learning Representations (ICLR’15)
- Li Siming, 2011, Proceedings of the 15th Conference on Computational Natural Language Learning. Association for Computational Linguistics, 220--228
- Lin Chin-Yew, 2004, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 8
- Liu Chenxi, 2017, Yuille
- Maddison Chris J., 2017, International Conference on Learning Representations (ICLR’17)
- Mao Junhua, 2015, International Conference on Learning Representations (ICLR’15)
- Mao Junhua, 2014, Yuille
- Mitchell Margaret, 2012, Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 747--756
- 10.5555/1873738.1873752 · DOI
- Mitchell Margaret, 2013, Kees Van Deemter, and Ehud Reiter
- 10.1145/1273496.1273577 · DOI
- 10.5555/645313.649410 · DOI
- Ordonez Vicente, 2011, Berg
- Ranzato Marc’Aurelio, 2016, International Conference on Learning Representations (ICLR’16)
- Zhou Ren Hailin Jin Zhe Lin Chen Fang and Alan Yuille. 2015. Multi-instance visual-semantic embedding. arXiv preprint arXiv:1512.06963. Zhou Ren Hailin Jin Zhe Lin Chen Fang and Alan Yuille. 2015. Multi-instance visual-semantic embedding. arXiv preprint arXiv:1512.06963. · arXiv
- Vadivel Karthikeyan Shanmuga, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3241--3250
- Simonyan Karen, 2015, International Conference on Learning Representations (ICLR’15)
- Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203. Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203. · arXiv
- Sutskever Ilya, 2014, Le
- van Deemter Kees, Proceedings of the 4th International Natural Language Generation Conference. Association for Computational Linguistics, 130--132
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998--6008.
- 10.5555/1708322.1708334 · DOI
- Wang Qingzhong, 2018, Chan
- Wang Yufei, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17)
- Yang Zhilin, 30th Conference on Neural Image Processing System (NIPS’16)
- Xu Kelvin, 2015, International Conference on Machine Learning. 2048--2057
- Yang Linjie, 2016, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16)
- Yu Lantao, 2017, Proceedings of the 31st AAAI Conference on Artificial Intelligence.
- Matthew, 2014, Zeiler and Rob Fergus
- Zhang Li, 2017, Hospedales
Cited by
Sorted by citations
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
“The use of deep learning for image captioning has been surveyed very recently [230], [231], and tasks that generate text given textual inputs are discussed in Sections IV-E–IV-G.”
BackgroundClimbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082
“One approach to providing grounding is to train distributional models on corpora augmented with perceptual data, such as photos (Hossain et al., 2019) or other modalities (Kiela and Clark, 2015; Kiela et al., 2015).”
BackgroundConvolutional neural networks in medical image understanding: a survey
Sarvamangala, D. R., & Kulkarni, R. V. (2021) · Evolutionary Intelligence · doi:10.1007/s12065-020-00540-3 · Cited by 1,043
“Researchers are also working on image captioning (textual representations of the image) [29].”
BackgroundZhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“video [179] while text-to-image generation aims to generate a realistic image that matches the given text description [180].”
BackgroundLearning Dynamics and Heterogeneity of Spatial-Temporal Graph Data for Traffic Forecasting
Guo, S., Lin, Y., Wan, H., Li, X., & Cong, G. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3056502 · Cited by 800
“It has been widely used and achieved great successes in various tasks such as natural language processing (NLP) [39], [40], image caption [41] and speech recognition [42].”
Background
Metadata
- Authors
- Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H.
- Affiliations
- Murdoch University (AU)
- Venue
- ACM Computing Surveys
- Published
- 2019-02-04
- Volume
- 51
- Issue
- 6
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2896348597
- Cited by
- 879
- References
- 166
- Citations by year
- 2026: 55 · 2025: 113 · 2024: 124 · 2023: 167 · 2022: 153 · 2021: 127 · 2020: 104 · 2019: 36
- FWCI
- 47.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Murdoch University Research Portal (submittedVersion)
- Funders
- Australian Research Council (DE120102960)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- image captioning, object recognition, sentence generation, evaluation metrics, deep learning architectures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
