Next-generation image captioning: A survey of methodologies and emerging challenges from transformers to Multimodal Large Language Models
Open accessAbstract
The widespread availability of visual data on the Internet has fueled a significant interest in image-to-text captioning systems. Automated image captioning remains a challenging multimodal analytics task, integrating advances in both Computer Vision (CV) and Natural Language Processing (NLP) to understand image content and generate semantically meaningful textual descriptions. Modern deep learning-based approaches have supplanted traditional approaches in image captioning, leading to more efficient and sophisticated models. The development of attention mechanisms and transformer-based architectures has further enhanced the modeling of both language and visual data. Despite these gains, challenges such as long-tailed object recognition, bias in training data, and shortcomings in evaluation metrics constrain the capabilities of current models. Furthermore, an important breakthrough has been made with the recent emergence of Multimodal Large Language Models (MLLMs). By incorporating textual and visual data, MLLMs provide improved captioning flexibility, generative capabilities, and reasoning. However, these models introduce new challenges, including faithfulness, grounding, and computational cost. Although relatively few studies have comprehensively surveyed these developments, this paper provides a thorough analysis of Transformer-based captioning approaches, investigates the shift to MLLMs, and discusses associated challenges and opportunities. We also present a performance comparison of the latest models on the MS-COCO benchmark and conclude with perspectives on potential future research directions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vision-language models often fall short when generating reliable alt text for technical figures, where precision, clarity, and conceptual correctness are essential.
“While early work on vision-language models has shown promising results on everyday images, these approaches often fall short when generating reliable alt text for technical figures, where precision, clarity, and conceptual correctness are essential [1].”
From Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study (2026) · cites this paper as [1]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
(2004) · Cited by 9,718 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
(2005) · Cited by 4,152 · see citing works
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,682 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
International Conference on Artificial Neural Networks
Boubacar, H. A., & Lecoeuche, S. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90065-s · Cited by 2,088 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
CoCa: Contrastive Captioners are Image-Text Foundation ModelsPreprint
Yu, J.-H., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y.-H. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.01917 · Cited by 1,899 · see citing works
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · see citing works
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityPreprint
Ye, Q.-H., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P.-C., Shi, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14178 · Cited by 1,318 · see citing works
Meshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259 · see citing works
2006 IEEE International Conference on Robotics and Automation
(2005) · IEEE Transactions on Robotics · doi:10.1109/tro.2005.847362 · Cited by 1,115 · see citing works
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
VinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 965 · see citing works
Proceedings Of The International Joint Conference On Neural Networks 2003
(2003) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2003.1223944 · Cited by 953 · see citing works
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works
GIT: A Generative Image-to-text Transformer for Vision and LanguagePreprint
Wang, J., Yang, Z.-Y., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z.-C., Liu, C., & Wang, L.-J. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.14100 · Cited by 817 · see citing works
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z.-Y., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C.-H., Yue, X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.15010 · Cited by 777 · see citing works
X-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720 · see citing works
Murray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2005) · Meeting of the Association for Computational Linguistics · Cited by 716 · see citing works
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J. S., Tian, J., Zhang, Y. S., Shi, Z., Jianping Fan, B., & He, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 614 · see citing works
ClipCap: CLIP Prefix for Image CaptioningPreprint
Mokady, R., Hertz, A., & Bermano, A. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09734 · Cited by 605 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · see citing works
Multimodal Transformer With Multi-View Visual Representation for Image Captioning
Yu, J., Li, J., Zhou, Y., & Huang, Q. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2947482 · Cited by 474 · see citing works
Wang, P., An, Y., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., & Yang, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03052 · Cited by 421 · see citing works
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · see citing works
Entangled Transformer for Image Captioning
Li, G., Zhu, L., Liu, P., & Yang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00902 · Cited by 406 · see citing works
Image Captioning: Transforming Objects into WordsPreprint
Herdade, S., Kappeler, A., Boakye, K., & Soares, J. C. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05963 · Cited by 395 · see citing works
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · see citing works
In Defense of Grid Features for Visual Question Answering
Jiang, H., Misra, I., Rohrbach, M., Miller, E. G., & Chen, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01028 · Cited by 379 · see citing works
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373 · see citing works
Dual-level Collaborative Transformer for Image Captioning
Luo, Y., Ji, J., Sun, X., Cao, L., Wu, Y., Huang, F., Lin, C.-W., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16328 · Cited by 343 · see citing works
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words
Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01521 · Cited by 297 · see citing works
Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Guo, L., Liu, J., Zhu, X., Yao, P., Lu, S., & Lu, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01034 · Cited by 259 · see citing works
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 241 · see citing works
PaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235 · see citing works
Ji, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16258 · Cited by 208 · see citing works
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 205 · see citing works
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and GenerationPreprint
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., & Zhou, M. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06353 · Cited by 187 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · see citing works
End-to-End Transformer Based Model for Image Captioning
Wang, Y., Xu, J., & Sun, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20160 · Cited by 169 · see citing works
GRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features
Nguyen, V.-Q., Suganuma, M., & Okatani, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_10 · Cited by 152 · see citing works
Injecting Semantic Concepts into End-to-End Image Captioning
Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01748 · Cited by 141 · see citing works
Hierarchical Attention Network for Image Captioning
Wang, W., Chen, Z., & Hu, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018957 · Cited by 140 · see citing works
Image Captioning Through Image Transformer
He, S., Liao, W., Tavakoli, H. R., Yang, M. Y., Rosenhahn, B., & Pugeault, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_10 · Cited by 140 · see citing works
Towards local visual modeling for image captioning
Ma, Y., Ji, J., Sun, X., Zhou, Y., & Ji, R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109420 · Cited by 139 · see citing works
Comprehending and Ordering Semantics for Image Captioning
Li, Y., Pan, Y., Yao, T., & Mei, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01746 · Cited by 138 · see citing works
Language Models with TransformersPreprint
Wang, C., Li, M., & Smola, A. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09408 · Cited by 130 · see citing works
CPTR: Full Transformer Network for Image CaptioningPreprint
Liu, W., Chen, S., Guo, L., Zhu, X., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10804 · Cited by 128 · see citing works
Dual Global Enhanced Transformer for image captioning
Xian, T., Li, Z., Zhang, C., & Ma, H. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.01.011 · Cited by 109 · see citing works
Cross on Cross Attention: Deep Fusion Transformer for Image Captioning
Zhang, J., Xie, Y., Ding, W., & Wang, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3243725 · Cited by 107 · see citing works
Captioning Transformer with Stacked Attention Modules
Zhu, X., Li, L., Liu, J., Peng, H., & Niu, X. (2018) · Applied Sciences · doi:10.3390/app8050739 · Cited by 102 · see citing works
Learning to Collocate Neural Modules for Image Captioning
Yang, X., Zhang, H., & Cai, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00435 · Cited by 101 · see citing works
Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
Kuo, C.-W., & Kira, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01744 · Cited by 99 · see citing works
Visuals to Text: A Comprehensive Review on Automatic Image Captioning
Ming, Y., Hu, N., Fan, C., Feng, F., Zhou, J., & Yu, H. L. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105734 · Cited by 80 · see citing works
S2 Transformer for Image Captioning
Zeng, P., Zhang, H., Song, J., & Gao, L. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/224 · Cited by 78 · see citing works
Deep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 78 · see citing works
Adaptive Path Selection for Dynamic Image Captioning
Xian, T., Li, Z., Tang, Z., & Ma, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3155795 · Cited by 75 · see citing works
DIFNet: Boosting Visual Information Flow for Image Captioning
Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., & Ji, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01749 · Cited by 74 · see citing works
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Hu, X., Yin, X. P., Lin, K., Zhang, L., Gao, J., Wang, L., & Liu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16249 · Cited by 73 · see citing works
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023) · neural information processing systems · doi:10.52202/075280-3138 · Cited by 72 · see citing works
Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
Wang, Y., Sun, F., Lu, M., & Yao, A. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413621 · Cited by 68 · see citing works
Double-Stream Position Learning Transformer Network for Image Captioning
Jiang, W., Zhou, W., & Hu, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3181490 · Cited by 65 · see citing works
Noise Augmented Double-Stream Graph Convolutional Networks for Image Captioning
Wu, L., Xu, M., Sang, L., Yao, T., & Mei, T. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3036860 · Cited by 63 · see citing works
Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image RepresentationsPreprint
Liu, F., Liu, Y., Ren, X., He, X., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06139 · Cited by 63 · see citing works
Adaptive Semantic-Enhanced Transformer for Image Captioning
Zhang, J., Fang, Z., Sun, H., & Wang, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3185320 · Cited by 56 · see citing works
Sharma, H., & Padha, D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10488-2 · Cited by 51 · see citing works
A statistical approach to secure health care services from DDoS attacks during COVID-19 pandemic
Zhou, Z., Gaurav, A., Gupta, B. B., Hamdi, H., & Nedjah, N. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06389-6 · Cited by 47 · see citing works
Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning
Hu, J. C., Cavicchioli, R., & Capotondi, A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386812 · Cited by 45 · see citing works
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training.
Hu, X., Yin, X. P., Lin, K., Wang, L., Zhang, L., Gao, J., & Liu, Z. (2020) · Cited by 42 · see citing works
HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning
Kuo, C.-W., & Kira, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01062 · Cited by 41 · see citing works
Sharma, D., Dhiman, C., & Kumar, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119773 · Cited by 38 · see citing works
Application of Deep Learning and Transfer Learning Techniques for Medical Image Classification
Sakirin, T., & Said, R. B. (2025) · EDRAAK · doi:10.70470/edraak/2025/006 · Cited by 34 · see citing works
A Review of Transformer-Based Approaches for Image Captioning
Ondeng, O., Absaloms, H. O., & Akuon, P. O. (2023) · Applied Sciences · doi:10.3390/app131911103 · Cited by 33 · see citing works
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability
Cornia, M., Baraldi, L., & Cucchiara, R. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196653 · Cited by 31 · see citing works
TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning
Fan, Z., Wei, Z., Wang, S., Wang, R., Li, Z., Shan, H., & Huang, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/91 · Cited by 29 · see citing works
ACORT: A compact object relation transformer for parameter efficient image captioning
Tan, J. H., Tan, Y. H., Chan, C. S., & Chuah, J. H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.081 · Cited by 26 · see citing works
Semantic Representations With Attention Networks for Boosting Image Captioning
Hafeth, D. A., Kollias, S., & Ghafoor, M. (2023) · IEEE Access · doi:10.1109/access.2023.3268744 · Cited by 26 · see citing works
Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning
Li, J., Mao, Z., Li, H., Chen, W., & Zhang, Y. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3638558 · Cited by 24 · see citing works
Image captioning using transformer-based double attention network
Parvin, H., Naghsh‐Nilchi, A. R., & Mohammadi, H. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106545 · Cited by 21 · see citing works
Efficient Image Captioning Based on Vision Transformer Models
Elbedwehy, S., Medhat, T., Hamza, T., & Alrahmawy, M. F. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.029313 · Cited by 19 · see citing works
Survey on Neural Networks in Networking: Applications and Advancements
Ayad, J. (2024) · Babylonian Journal of Networking · doi:10.58496/bjn/2024/014 · Cited by 18 · see citing works
Comparative Study of Transformer and LSTM Network with Attention Mechanism on Image Captioning
Dandwate, P., Shahane, C., Jagtap, V. S., & Karande, S. C. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-3761-5_47 · Cited by 18 · see citing works
Context-aware transformer for image captioning
Yang, X. S., Wang, Y., Chen, H., Li, J., & Huang, T. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126440 · Cited by 16 · see citing works
Multi-Modal Graph Aggregation Transformer for image captioning
Chen, L., & Li, K. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106813 · Cited by 15 · see citing works
ETransCap: efficient transformer for image captioning
Mundu, A., Singh, S. K., & Dubey, S. (2024) · Applied intelligence (Boston) · doi:10.1007/s10489-024-05739-w · Cited by 14 · see citing works
TSFNet: Triple-Steam Image Captioning
Hu, N., Ming, Y., Fan, C., Feng, F., & Lyu, B. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3215861 · Cited by 13 · see citing works
Alsayed, A., Arif, M., Qadah, T. M., & Alotaibi, S. S. (2023) · Applied Sciences · doi:10.3390/app131910894 · Cited by 13 · see citing works
Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning
Hafeth, D. A., & Kollias, S. (2024) · Sensors · doi:10.3390/s24061796 · Cited by 13 · see citing works
Full-Memory Transformer for Image Captioning
Lu, T., Wang, J., & Min, F. (2023) · Symmetry · doi:10.3390/sym15010190 · Cited by 13 · see citing works
Multi-feature fusion enhanced transformer with multi-layer fused decoding for image captioning
Zhang, J., Fang, Z., & Wang, Z. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04202-y · Cited by 12 · see citing works
A Survey on Attention-Based Models for Image Captioning
Osman, A. A. E., Shalaby, M. A. W., Soliman, M., & Elsayed, K. M. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140249 · Cited by 12 · see citing works
Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
Bucciarelli, D., Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92089-9_22 · Cited by 11 · see citing works
Wu, D., Li, H., Gu, C., Guo, L., & Liu, H. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548223 · Cited by 11 · see citing works
From Templates to Transformers: A survey of Multimodal Image Captioning Decoders
Sharma, H., & Padha, D. (2023) · International Conference on Computer, Electronics & Electrical Engineering & their Applications (IC2E3) · doi:10.1109/ic2e357697.2023.10262494 · Cited by 9 · see citing works
A Survey on Automatic Image Captioning Approaches: Contemporary Trends and Future Perspectives
Salgotra, G., Abrol, P., & Selwal, A. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10190-8 · Cited by 9 · see citing works
Dual visual align-cross attention-based image captioning transformer
Ren, Y., Zhang, J., Xu, W., Lin, Y., Fu, B., & Thanh, D. N. H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19315-4 · Cited by 8 · see citing works
Transformer Approaches in Image Captioning: A Literature Review
Tsaniya, H., Fatichah, C., & Suciati, N. (2022) · doi:10.1109/icitee56407.2022.9954086 · Cited by 8 · see citing works
Attention-Based Deep Learning Model for Image Captioning: A Comparative Study
Khaing, P. P., & Yu, M. T. (2019) · International Journal of Image Graphics and Signal Processing · doi:10.5815/ijigsp.2019.06.01 · Cited by 8 · see citing works
Improve Image Captioning Via Relation Modeling
Huang, F., & Li, Z. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747820 · Cited by 7 · see citing works
Image Captioning In the Transformer AgePreprint
Xu, Y., Li, L., Xu, H., Huang, S., Huang, F., & Cai, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07374 · Cited by 7 · see citing works
Transformer based Multitask Learning for Image Captioning and Object Detection
Basak, D., Srijith, P. K., & Desarkar, M. S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2253-2_21 · Cited by 6 · see citing works
Triple-level relationship enhanced transformer for image captioning
Zheng, A., Zheng, S., Bai, C., & Chen, D. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01073-2 · Cited by 3 · see citing works
Attending to Transforms: A Survey on Transformer-based Image Captioning
Ambilduke, K., Jayakumar, T., Farooqui, L., Padole, H. P., & Singh, A. (2023) · International Conference on Paradigm Shifts in Communications Embedded Systems, Machine Learning and Signal Processing (PCEMS) · doi:10.1109/pcems58491.2023.10136098 · Cited by 3 · see citing works
A Novel Cross-Fusion Method of Different Types of Features for Image Captioning
Lou, L., Lü, K., & Xue, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191971 · Cited by 3 · see citing works
Transformer with multi-level grid features and depth pooling for image captioning
Bui, D. C., Nguyen, T., & Nguyen, K. T. T. M. (2024) · Machine Vision and Applications · doi:10.1007/s00138-024-01599-z · Cited by 3 · see citing works
Transformer with token attention and attribute prediction for image captioning
Song, L., Wang, Y., Shi, L., Yu, J., Li, F., & Xiang, S. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.11.022 · Cited by 3 · see citing works
Skipped-Connection Transformer for Image Captioning
Lou, L., Lu, K., & Xue, J. (2022) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr54900.2022.00036 · Cited by 2 · see citing works
Feature-Fusion Parallel Decoding Transformer for Image Captioning
Zhu, C., Xia, Y., & Lu, Q. (2022) · Academic Journal of Science and Technology · doi:10.54097/ajst.v2i1.905 · Cited by 2 · see citing works
A Comprehensive Review on Automatic Image Captioning Using Deep Learning
Kavitha, P. V., & Karpagam, V. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-2177-3_17 · Cited by 2 · see citing works
Improving Image Captioning with Feature Filtering and Injection
Guo, M.-H., Chen, Q., Fang, X., Bao, J., & Xiang, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44210-0_30 · Cited by 1 · see citing works
Verma, N. K., & Kene, P. (2023) · International Conference on Developments in eSystems Engineering (DeSE) · doi:10.1109/dese60595.2023.10469642 · Cited by 1 · see citing works
Improving Image Captioning via Dual-Stream Multi-Layer Dynamic FusionPreprint
Wang, R., Jia, W., Yang, J., & Xue, L. (2023) · Research Square · doi:10.21203/rs.3.rs-2794281/v1 · Cited by 1 · see citing works
Distance-aware multilayer aggregation transformer for image captioning
Xu, H., Xue, L., Wang, R., & Yang, J. (2024) · International Conference on Machine Vision, Automatic Identification, and Detection (MVAID) · doi:10.1117/12.3035618 · Cited by 1 · see citing works
HiFi-Score: Fine-Grained Image Description Evaluation with Hierarchical Parsing Graphs
Yao, Z., Wang, R., & Chen, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73033-7_25 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 138 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
HPRNet : A holistic position-aware residual network for image captioning
Zhao, Y., Tao, Q., Sun, P., Huang, J., Song, W., & Hu, N. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113249 · Cited by 4 · see citing works
Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study
Cardia, M., Angileri, L., Poggianti, C., & Leporini, B. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800457 · Cited by 2 · see citing works
Zhong, P., Hu, W., Zhao, Y., & Zhang, F. (2026) · npj Heritage Science · doi:10.1038/s40494-026-02343-8 · Cited by 1 · see citing works
CLIP-AMR-GPT: Enhancing Image Captioning via Cross-Modal Semantics Fusion and GPT-Based Re-ranking
Thịnh, N. V., Lang, T. V., & Hải, N. M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4960-3_17
Yue, W., Chang, K., Liu, X., Tan, K., & Chen, W. (2025) · Symmetry · doi:10.3390/sym17122070
Beyond text: Multimodal stance detection in Arabic tweets
AlShenaifi, N., & Alangari, N. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100823
Mulyono, S., Adi, K., & Sugiharto, A. (2025) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.1109/eecsi67060.2025.11290311
Progressive Fine-Tuning for Robust Multimodal Image Captioning
Wang, S., Yang, S., & Yang, Y. (2025) · International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice68195.2025.11382256
Raj, D. S. P., Anuradha, D. G., Kavitha, V., Shalini, K., Steffi, R., & Mathew, D. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-17300-3_14 · Cited by 2 · see citing works
Sanjana, B., Ravi, A., Patnaik, V. S., Praneel, A. S. V., Roy, R., & Rao, D. T. S. (2025) · International Conference on Artificial Intelligence and Speech Technology (AIST) · doi:10.1109/aist68591.2025.11441567
Dotsformer: Capturing Chain-Loop Structures for Transformer in Dots-and-Boxes
Zhang, R., Xu, C., Wu, K., Zheng, M., Liu, X., & Wang, J. (2026) · Applied Sciences · doi:10.3390/app16073395
Phuong, H. S., Nguyen, X., Pham, P. H., Le, D. M., Do, D. K., Ngo, T. D., Pham, Q. V., & Le, M. (2025) · Asia Meeting on Environment and Electrical Engineering (EEE-AM) · doi:10.1109/eee-am66675.2025.11473622
LDCAP: Robust Image Captioning via Latent Compression and Dynamic Decoder ConditioningPreprint
Reddy, V., Poola, S., shaik, M., kistaparapu, S. D., & sajja, V. (2026) · Research Square · doi:10.21203/rs.3.rs-9480848/v1
Secure Image Captioning Framework using BLIP-2 and Blockchain
Charishma, S., Bhargavi, T., & Bhaskaru, O. (2026) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca69024.2026.11570568
Domain-Specific EfficientNet-Transformer based Vision Language Model for Campus Navigation
Mayya, A., Khaire, P., & Zaki, M. (2025) · IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei68356.2025.11603679
Alzahrani, S., Mohammad, F., & Hussain, N. (2026) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.416655
Automatic Image Captioning for Project Documentation at the ITE
Hamed, O., Zaidkilani, N., Rosenthal, E., & Natour, G. (2026) · International Conference on Federated Learning and Intelligent Computing Systems (FLICS) · doi:10.1109/flics70075.2026.11621954
ISAR-Mamba: A Dual-Stream Gated Mamba with Hierarchical Spatial Summaries for ISAR Image Captioning
He, Y., Pan, A., Li, Y., Wang, J., Qu, W., Zhu, W., Ji, W., Tang, G., & Lv, J. (2026) · Sensors · doi:10.3390/s26185916
Xie, Y., & Saedon, M. A. M. (2026) · International Conference on Artificial Intelligence, Digital Media and Education · doi:10.1145/3839592.3839713
Multi-level Knowledge Distillation for Lightweight Image Captioning
Thịnh, N. V., Khiem, N. T., Hoa, N. T. N., Man, D. T., & Yen, N. H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6510-7_22
- Authors
- Abdulgalil, H. D., & Basir, O. A.
- Affiliations
- University of Waterloo (CA)
- Corresponding author
- Abdulgalil, Huda Diab
- Venue
- Natural Language Processing Journal
- Publisher
- Elsevier BV
- Published
- 2025-09
- Volume
- 12
- Pages
- 100159
- ISSN
- 2949-7191 (ISSN-L)
- OpenAlex
- W4411176006
- Cited by
- 20
- References
- 138
- Citations by year
- 2026: 12 · 2025: 8
- FWCI
- 7.07
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Natural Language Processing Journal (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- image captioning, multimodal large language models, attention mechanism, training data bias, grounding, faithfulness, computational cost, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
