The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Cited by
Sorted by citations
Items 401–500 of 1,002
Sun, W., Wang, C., Wu, H., Miao, Y., Zhu, H., Wang, G., & Li, J. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108460 · Cited by 12
Exploring better image captioning with grid features
Yan, J., Xie, Y., Guo, Y., Wei, Y., & Luan, X. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01341-8 · Cited by 14
“The comparison methods include Up-Down [33], GCN-LSTM [52], SGAE [53], AoANet [54], Transformer [32], X-Transformer [31], and RSTNet [14].”
Noise-Aware Image Captioning with Progressively Exploring Mismatched Words
Fu, Z., Song, K., Zhou, L., & Yang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29097 · Cited by 15
“More-over, as a generalized framework, NIC can be applied to any existing image captioning models, and several typical approaches (Huang et al. 2019; Zhang et al. 2021; Luo et al. 2021; Wang, Xu, and Sun 2022) are adopted for validation.”
BackgroundMethodInfluentialShow, tell and rectify: Boost image caption generation via an output rectifier
Ge, G., Han, Y., Hao, L., Hao, K., Wei, B., & Tang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127651 · Cited by 12
Toward Attribute-Controlled Fashion Image Captioning
Cai, C., Yap, K., & Wang, S. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3671000 · Cited by 12
“Since our proposed ACC model is an end-to-end captioning method, we followed the settings in recent end-to-end transformer-based captioning works [38, 66].”
Efficiently Gluing Pre-Trained Language and Vision Models for Image Captioning
Song, P., Zhou, Y., Yang, X., Liu, D., Hu, Z., Wang, D., & Wang, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3682067 · Cited by 12
“Following the standard practice in image captioning [6, 16], the weights in the connector are trained in two stages.”
InfluentialSharma, A., & Aggarwal, M. (2025) · SN Computer Science · doi:10.1007/s42979-025-03719-6 · Cited by 12
Attribute-driven image captioning via soft-switch pointer
Zhou, Y., Long, J., Xu, S., & Shang, L. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.08.021 · Cited by 12
What Happens in Crowd Scenes: A New Dataset About Crowd Scenes for Image Captioning
Wang, L., Li, H., Hu, W., Zhang, X., Qiu, H., Meng, F., & Wu, Q. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3192729 · Cited by 13
Multi-feature fusion enhanced transformer with multi-layer fused decoding for image captioning
Zhang, J., Fang, Z., & Wang, Z. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04202-y · Cited by 12
“…including: Adaptive [22], Bi-DPN [24], SCST:Att2all [26], Stack-Cap [27], Stack-comLL [28], MAGAN [29], Bottom-up [7], ASIA [25], VORN [42], AoANet [44], ICTIT [43], M 2 -Transformer [45], LSTM-A 3 [32], VS-LSTM [33], pLSTM-A-2 ∗ [40], GCN-LSTM [8], SGAE [38], VSUA [39], and EnTangled-T [9].”
MethodResultInfluentialContextual and selective attention networks for image captioning
Wang, J., Li, Y., Pan, Y., Yao, T., Tang, J., & Mei, T. (2022) · Science China Information Sciences · doi:10.1007/s11432-020-3523-6 · Cited by 13
StyleBabel: Artistic Style Tagging and Captioning
Ruta, D., Gilbert, A., Aggarwal, P., Marri, N., Kale, A., Briggs, J., Speed, C., Jin, H., Faieta, B., Filipkowski, A., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_13 · Cited by 12
A Survey on Attention-Based Models for Image Captioning
Osman, A. A. E., Shalaby, M. A. W., Soliman, M., & Elsayed, K. M. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140249 · Cited by 12
“[86] proposed Attention on Attention (AoA) approach, which adds attention over the traditional attention.”
BackgroundExploring region features in remote sensing image captioning
Zhao, K., & Xiong, W. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103672 · Cited by 13
A Sparse Transformer-Based Approach for Image Captioning
Zhou, L., Zhou, C., Chen, S., Huang, Y., & Liu, X. (2020) · IEEE Access · doi:10.1109/access.2020.3024639 · Cited by 12
SentiStory: A Multi-Layered Sentiment-Aware Generative Model for Visual Storytelling
Chen, W., Liu, X., & Niu, J. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3183648 · Cited by 13
“The attention mechanism can be applied to various types of data, such as spatial grids [3], [19], semantic metadata [4], [20], [21], and object region proposals [2], [16]–[18], [22].”
BackgroundAttribute assisted teacher-critical training strategies for image captioning
Huang, Y., Chen, J., Ma, H., Ma, H., Ouyang, W., & Yu, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.07.068 · Cited by 14
Self-citationAttacking Image Captioning Towards Accuracy-Preserving Target Words Removal
Ji, J., Sun, X., Zhou, Y., Ji, R., Chen, F., Liu, J., & Tian, Q. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414009 · Cited by 12
“Two-stage models consist of a bottom-up module and a top-down module, where the bottom-up module first uses an object detector to detect multiple regions in the image, and then the top-down module attends to the most relevant detected regions to generate captions [2, 16, 21, 22].”
MethodGeometry-sensitive semantic modeling in visual and visual-language domains for image captioning
Zhu, W., Jiang, Z., & He, Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110330 · Cited by 11
Question-controlled Text-aware Image Captioning
Hu, A., Chen, S., & Jin, Q. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475452 · Cited by 11
Image Caption and Medical Report Generation Based on Deep Learning: a Review and Algorithm Analysis
Li, R., Wang, Z., & Zhang, L. (2021) · International Conference on Computer Information Science and Artificial Intelligence (CISAI) · doi:10.1109/cisai54367.2021.00078 · Cited by 11
DFEN: Dual Feature Enhancement Network for Remote Sensing Image Caption
Zhao, W., Yang, W., Chen, D., & Wei, F. (2023) · Electronics · doi:10.3390/electronics12071547 · Cited by 11
A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering
Al-Hadhrami, S. A., Menaï, M. E. B., Al-Ahmadi, S. A., & Alnafessah, A. (2023) · IEEE Access · doi:10.1109/access.2023.3335216 · Cited by 14
“Building on Huang et al. [101], Rahman et al. used the Attention on Attention (AoA) module to resolve this limitation.”
MethodStyle-Enhanced Transformer for Image Captioning in Construction Scenes
Song, K., Chen, L., & Wang, H. (2024) · Entropy · doi:10.3390/e26030224 · Cited by 11
“As shown in Table 2, our research method achieves higher accuracy on the MSCOCO test set compared with the models Up-down [18], AoANet [22], X-LAN [19] and ViTCAP [43] models.”
BackgroundMethodResultInfluentialCycle-Consistency Learning for Captioning and Grounding
Wang, N., Deng, J., & Jia, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28363 · Cited by 11
Regular Constrained Multimodal Fusion for Image Captioning
Wang, L., Chen, H., Liu, Y., & Lyu, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3425513 · Cited by 13
Mining core information by evaluating semantic importance for unpaired image captioning
Wei, J., Li, Z., Zhang, C., & Ma, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106519 · Cited by 11
Vision-language pre-training via modal interaction
Cheng, H., Ye, H., Zhou, X., Liu, X., Chen, F., & Wang, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110809 · Cited by 12
Multi-granularity semantic relational mapping for image caption
Gao, N., Yao, R., Chen, P., Liang, R., Sun, G., & Tang, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125847 · Cited by 12
Intra- and Inter-Head Orthogonal Attention for Image Captioning
Zhang, X., Jia, A., Ji, J., Qu, L., & Ye, Q. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3528216 · Cited by 13
“In image captioning, attention model [2], [20] and its variations [21], [22], [23], [24], [25] focus on better improving the alignment of objects and words.”
Multi-level semantic-aware transformer for image captioning
Xu, Q., Song, S., Wu, Q., Jiang, B., Luo, B., & Tang, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107390 · Cited by 11
Improving image captioning descriptiveness by ranking and LLM-based fusion
Celona, L., Bianco, S., Donzella, M., & Napoletano, P. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11672-x · Cited by 41
“Given N SoTA image captioning models and an image I , it is possible to generate N different captions { T 1 , ..., T N } that describe the content of the image.”
Sun, J.-Y., Jung, S., & Ko, S.-J. (2020) · IEEE Access · doi:10.1109/access.2020.3001679 · Cited by 11
Gaze estimation via bilinear pooling-based attention networks
Ren, D., Chen, J., Zhong, J., Lu, Z., Jia, T., & Li, Z. (2021) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2021.103369 · Cited by 11
Self-citationDual Position Relationship Transformer for Image Captioning
Wang, Y., Qian, W., Nie, R., Xu, D., Cao, J., & Kim, P. (2022) · Big Data · doi:10.1089/big.2021.0262 · Cited by 11
“For example, Huang et al. 29 proposed an ‘‘Attention on Attention’’ module, which used the sinusoidal position encoding method for text embedding.”
Hadamard Product Perceptron Attention for Image Captioning
Jiang, W., & Hu, H. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10980-w · Cited by 11
“Hence, this framework still suffers from insufficient use of visual features [6, 15].”
BackgroundMethodInfluentialSupervised Deep Learning Techniques for Image Description: A Systematic Review
López-Sánchez, M., Hernández-Ocaña, B., Chávez-Bosquez, O., & Hernández-Torruco, J. (2023) · Entropy · doi:10.3390/e25040553 · Cited by 14
“[39] proposed a method in which they use a module called “attention on attention” (AoA), allowing them to determine the relevance between the results for attention and consultation.”
Lightweight Transformer with GRU Integrated Decoder for Image Captioning
Sharma, D., Dingliwal, R., Dhiman, C., & Kumar, D. (2022) · International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) · doi:10.1109/sitis57111.2022.00072 · Cited by 11
“A family of attention-based methods [5] [6] [7], aimed to link the predicted lexicons to specific regions in the image.”
MethodResultSketch Captioning Using LSTM and BiLSTM
Dahikar, Y., Chikmurge, D. V., & Kharat, S. (2023) · International Conference on Networks, Multimedia and Information Technology (NMITCON) · doi:10.1109/nmitcon58196.2023.10275867 · Cited by 11
“The development of neural machine translation [4] and [6] served as inspiration for them.”
BackgroundSentiment-Oriented Transformer-Based Variational Autoencoder Network for Live Video Commenting
Fu, F., Fang, S., Chen, W., & Mao, Z. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3633334 · Cited by 12
Image captioning via proximal policy optimization
Zhang, L., Zhang, Y., Zhao, X., & Zou, Z. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104126 · Cited by 11
A Survey on Recent Advances in Image Captioning
Chen, F., Li, X., Tang, J., Li, S., & Wang, T. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1914/1/012053 · Cited by 11
“[12] further presented an attention-on-attention (AoA) module to model the relevance between the attention result and the attention query.”
Generate classical Chinese poems with theme-style from images
Wu, C., Wang, J., Yuan, S., Wang, L., & Zhang, W. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.05.016 · Cited by 12
Robotic Indoor Scene Captioning from Streaming Video
Li, X., Guo, D., Liu, H., & Sun, F. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560904 · Cited by 11
“Although the existing image captioning methods [8] [9] [10] are available to generate brief captions for the indoor scene, they cannot provide informative captions that contain enough appeared objects and the exact number of the objects.”
MethodTowards a large-scale person search by vietnamese natural language: dataset and methods
Pham, T. T. T., Nguyen, H.-Q., Phan, H., Do, T.-N.-D., Nguyen, T.-B., Tran, T.-H., & Le, T. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12138-1 · Cited by 11
“Moreover, in order to model the relationship between image regions and words, attention mechanism and graph-like structures are integrated into RNN [10, 54].”
MethodInfluentialContrastive semantic similarity learning for image captioning evaluation
Zeng, C., Kwong, S., Zhao, T., & Wang, H. (2022) · Information Sciences · doi:10.1016/j.ins.2022.07.142 · Cited by 13
CapWAP: Captioning with a PurposePreprint
Fisch, A., Lee, K., Chang, M.-W., Clark, J. H., & Barzilay, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.04264 · Cited by 11
“Huang et al. (2019) directly optimizes the CIDEr metric with policy gradients, while Luo et al. (2018) optimizes both CIDEr and a “discrimination” loss intended to encourage models to describe each image’s uniquely identifying aspects.”
MethodA comprehensive survey on deep learning approaches for image captioning: a systematic review
Al-Malla, M. A., Hamdoun, O., & Ghneim, N. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01377-w · Cited by 11
Improving Image Captioning via Enhancing Dual-Side Context Awareness
Gao, Y., Wang, N., Suo, W., Sun, M., & Wang, P. (2022) · International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531379 · Cited by 10
“Inspired by neural machine translation [29], existing image captioning methods [2, 13, 14, 25, 32] mainly adopt an encoder-decoder pipeline, where the former is responsible for generating contextual representation for the given image, the latter generates target words by combining both the…”
BackgroundMethodInfluentialGenerating Image Descriptions of Rice Diseases and Pests Based on DeiT Feature Encoder
Ma, C., Hu, Y., Liu, H., Huang, P., Zhu, Y., & Dai, D. (2023) · Applied Sciences · doi:10.3390/app131810005 · Cited by 10
IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer
Fan, K., Cai, X., & Niranjan, M. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8076-5_5 · Cited by 11
Wahid, A., Breslin, J. G., & Ali, M. I. (2024) · Human-Centric Intelligent Systems · doi:10.1007/s44230-023-00060-0 · Cited by 13
“Attention mechanisms have demonstrated their ability to boost performance in various areas, including speech recognition [36], machine translation [32], and image captioning [37].”
Triple-Stream Commonsense Circulation Transformer Network for Image Captioning
Li, J., Zhou, W., Wang, K., & Hu, H. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104165 · Cited by 10
Act Like a Radiologist: Radiology Report Generation Across Anatomical Regions
Chen, Q., Xie, Y., Wu, B., Chen, X., Ang, J. S. K., To, M., Chang, X., & Wu, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0960-4_3 · Cited by 15
“In this part, we provide more discussions, including the effect of different feature extractors in Section A.1, the impact of hyper-parameter λ in Section A.2, and whether feeding image tags into the model would cause information leakage in Section A.3.”
Hossain, M. S., Aktar, S., Liu, W., Gu, N., & Huang, Z. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01608-1 · Cited by 10
“Advancements in image captioning owe much to the development of encoder-decoder frameworks [7] and sophisticated attention mechanisms [8, 9], which enhance the generation of contextually rich and accurate descriptions.”
ORID: Organ-Regional Information Driven Framework for Radiology Report Generation
Gu, T., Yang, K., An, X., Feng, Z., Lin, D., & Cai, W. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00047 · Cited by 11
Tables to LaTeX: structure and content extraction from scientific tables
Kayal, P., Anand, M., Desai, H., & Singh, M. P. (2022) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-022-00420-9 · Cited by 10
“We propose models that utilize the attention-on-attention mechanism [14] (initially proposed for LSTM models) for image-to-token sequence translation.”
MethodSI-Net: spatial interaction network for deepfake detection
Wang, J., Du, X., Yu, C., Sun, Y., & Tang, J. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01114-w · Cited by 10
“After that, self-attention mechanism had also been widely used in various vision tasks, such as action recognition [16], object detection [6], image segmentation [60], image captioning [23], image restoration [63], image denoising [22], etc.”
MethodEgoCap and EgoFormer: First-Person Image Captioning with Context Fusion
Dai, Z., Tran, V., Markham, A., Trigoni, N., Rahman, M. A., Wijayasingha, L. N., Stankovic, J. A., & Li, C. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.03.012 · Cited by 11
Visual Relation of Interest Detection
Yu, F., Wang, H., Ren, T., Tang, J., & Wu, G. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413566 · Cited by 10
“Image captioning [1, 9, 33, 41] aims to represent the main content of an image with a complete sentence that might imply VROIs.”
BackgroundCagrandi, M., Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463587 · Cited by 10
“Recent captioning literature has shown that object regions are the leading solution to encode visual inputs [4, 37, 39], followed by self-attentive layers to model region relationships [11, 13, 16, 25, 27, 32].”
BackgroundImprove Image Captioning by Estimating the Gazing Patterns from the Caption
Alahmadi, R., & Hahn, J. K. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00251 · Cited by 11
“Additionally, transformer [40] has been utilized for image captioning either as image encoder [19] or as a language modeling [9].”
BackgroundMethodEfficient Modeling of Future Context for Image Captioning
Fei, Z. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547840 · Cited by 11
“In recent years, a large number of neural systems have been proposed for the image captioning task [3, 9, 16, 22, 24, 40, 53, 58].”
BackgroundMethodChen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2021) · arXiv (Cornell University) · Cited by 11
Self-citationNeural module networks: A review
Fashandi, H. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126518 · Cited by 11
Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment
Yu, L., Nikandrou, M., Jin, J., & Rieser, V. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/697 · Cited by 11
Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization
Hu, W., Qiao, L., Kang, W., & Shi, X. (2023) · Journal of Imaging · doi:10.3390/jimaging9080162 · Cited by 10
“Quantitative Evaluation We conducted experiments on the Thangka dataset and COCO dataset to evaluate the effectiveness of the proposed SCAMF-Net method and compared it with several existing mainstream image captioning models, including Bottom-Up and Top-Down Attention (BUTD) [13], Attention on Attention (AoA) [43], Ob…”
MethodParallel Dense Video Caption Generation with Multi-Modal Features
Huang, X., Chan, K., Ke, W., & Sheng, H. (2023) · Mathematics · doi:10.3390/math11173685 · Cited by 9
“Inspired by image captioning methods, such technology can be directly extended to video captioning, enabling the discovery of correlations between image sequences [16–18].”
Integrating grid features and geometric coordinates for enhanced image captioning
Zhao, F., Yu, Z., Zhao, H., Wang, T., & Bai, T. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05198-9 · Cited by 9
“Previous representative theses [6, 10, 43, 52] and the top three image caption models developed thus far [42, 45, 53] are compared with ours.”
Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder
Mohamed, A., Alwajih, F., Nagoudi, E. M. B., Inciarte, A. A., & Abdul-Mageed, M. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.1 · Cited by 14
Multi-Level Objective Alignment Transformer for Fine-Grained Oral Panoramic X-Ray Report Generation
Gao, N., Yao, R., Liang, R., Chen, P., Liu, T., & Dang, Y. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3368922 · Cited by 9
“Traditional methods, like [17], [18], [25], [26], [27], [28], [29], [30], [32] use the self-attention modules to model the features of different objects.”
MethodSet of Diverse Queries With Uncertainty Regularization for Composed Image Retrieval
Xu, Y., Wei, J., Bin, Y., Yang, Y., Ma, Z., & Shen, H. T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3401006 · Cited by 10
“Depending on the different natures of downstream tasks [49], [50], [51], [52], there are various types of model architecture choices for the vision-language pre-training (VLP).”
BackgroundMulti-Scale Features Are Effective for Multi-Modal Classification: An Architecture Search Viewpoint
Fu, P., Liang, X., Qian, Y., Guo, Q. J., Zhang, Y., Huang, Q., & Tang, K. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3470996 · Cited by 9
“…multi-modal methods, such as the multi-modal BERT series [39], the multi-head attention fusion method proposed by DFAF [40], and the novel attention fusion method called AoA introduced by AoANet [41], have achieved impressive results and become typical examples in the field of multi-modal fusion.”
RSIC-GMamba: A State-Space Model With Genetic Operations for Remote Sensing Image Captioning
Meng, L., Wang, J., Huang, Y., & Xiao, L. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3548664 · Cited by 9
“Historically, numerous methods [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15] based on the encoder–decoder paradigm have been proposed for natural image captioning (NIC).”
Song, S., Tang, H., Yang, H., & Li, X. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3591364 · Cited by 13
Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning
Guo, D., Lu, R., Chen, B., Zeng, Z., & Zhou, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01624-6 · Cited by 10
“RNN-based LMs have been successfully used in image paragraph captioning systems (discussed above), while single-sentence Transformer-based image captioning systems have started to attract attentions (25, 26, 21), not to mention the research on the image paragraph captioning.”
BackgroundMethodEAES: Effective Augmented Embedding Spaces for Text-Based Image Captioning
Nguyen, K. T. T. M., Bui, D. C., Trinh, T., & Vo, N. D. (2022) · IEEE Access · doi:10.1109/access.2022.3158763 · Cited by 9
“However, previous models just utilized only visual entities [2, 3, 4] or global semantic of the image [5], they completely ignored scene texts.”
BackgroundMethodKnowledge-Based Scene Graph Generation with Visual Contextual Dependency
Zhang, L., Yin, H., Hui, B., Liu, S., & Zhang, W. (2022) · Mathematics · doi:10.3390/math10142525 · Cited by 9
Position-guided transformer for image captioning
Hu, J., Yang, Y., Lu, Y., An, Y., & Pan, L. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104575 · Cited by 9
Mask-guided network for image captioning
Lim, J. H., & Chan, C. S. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.07.013 · Cited by 9
CropCap: Embedding Visual Cross-Partition Dependency for Image Captioning
Wang, B., Zhang, Z., Zhao, S., Zhang, H., Hong, R., & Wang, M. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612245 · Cited by 9
Injecting Prior Knowledge into Image Caption Generation
Goel, A., Fernando, B., Nguyen, T.-S., & Bilen, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66096-3_26 · Cited by 10
“Attention on attention is used in recent work [19].”
MethodResultInfluentialImage Caption Generation via Unified Retrieval and Generation-Based Method
Zhao, S., Li, L., Peng, H., Yang, Z., & Zhang, J. (2020) · Applied Sciences · doi:10.3390/app10186235 · Cited by 9
“[26] proposed an “Attention on Attention” module to model the relationships of different objects of the image on encoder, and refining for the decoder with self-attention mechanism.”
MethodNeural Symbolic Representation Learning for Image Captioning
Wang, X., Ma, L., Fu, Y., & Xue, X. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463637 · Cited by 9
S2TD: A Tree-Structured Decoder for Image Paragraph Captioning
Shi, Y., Liu, Y., Feng, F., Li, R., Ma, Z., & Wang, X. (2021) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3469877.3490585 · Cited by 9
“Although significant results have been achieved in single sentence captioning, such as [1, 5, 7, 9, 15, 21, 22, 33, 35, 36], generating a paragraph remains a challenge.”
BackgroundOCR-oriented Master Object for Text Image Captioning
Tang, W., Hu, Z., Song, Z., & Hong, R. (2022) · International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531431 · Cited by 9
“We compare the performance of our method with conventional image captioning models (Up-down [3] and AoANet [9]) and text image captioning models (M4C-Captioner [20], and MMA-SR [26]).”
BackgroundResultLearning to Collocate Visual-Linguistic Neural Modules for Image Captioning
Yang, X., Zhang, H., Gao, C., & Cai, J. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01692-8 · Cited by 9
“We compared CVLNM with the following state-of-the-art image captioners: SCST [59], UpDown [2], NBT [46], CAVP [85], RFNet [26], LBPF [57], GCN-LSTM [79], SGAE [76], HIP [80], ETA [36], ToW [19] and AoANet [24].”
MethodICAF: Iterative Contrastive Alignment Framework for Multimodal Abstractive Summarization
Zhang, Z., Shu, C., Chen, Y., Xiao, J., Zhang, Q., & Zheng, L. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892884 · Cited by 10
“, cross-modal retrieval [20], hard triplet loss [21] is used to enhance the corresponding representation across vision and language.”
MethodAutoCaption: Image Captioning with Neural Architecture SearchPreprint
Zhu, X., Wang, N., Guo, L., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09742 · Cited by 11
“Some works [18,35] have shown that image captioning model with the transformer has shown better performance than the model with the LSTM, but this will further bring a lot of computational overhead.”
BackgroundUniversal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style SeparationPreprint
Cornia, M., Baraldi, L., Fiameni, G., & Cucchiara, R. (2021) · arXiv (Cornell University) · Cited by 9
Dual visual align-cross attention-based image captioning transformer
Ren, Y., Zhang, J., Xu, W., Lin, Y., Fu, B., & Thanh, D. N. H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19315-4 · Cited by 8
“Huang et al. [29] introduced an extension to attention operators, where the final engaged information is weighted by context-guided gates.”
Cross-region feature fusion with geometrical relationship for OCR-based image captioning
Zhou, J., Yang, C. Z., Zhu, Y., & Zhang, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128197 · Cited by 9
Lee, H., Song, J., Hwang, K., & Kang, M. (2024) · IEEE Access · doi:10.1109/access.2024.3487588 · Cited by 8
“To generate accurate captions, studies are using algorithms such as VGG16, Inception-v3, RNN, GRU, and incorporating new techniques like attention mechanisms [49], [50], [51], [52], [53].”
MethodVisual Rotated Position Encoding Transformer for Remote Sensing Image Captioning
Liu, A., Meng, L., & Xiao, L. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3487846 · Cited by 9
“5) AoANet [62]: It introduces an attention mechanism called “attention on attention (AoA).””
Image Captioning Using Multimodal Deep Learning Approach
Farkh, R., Oudinet, G., & Foued, Y. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.053245 · Cited by 8
Long short-term memory autoencoder based network of financial indices
Tuhin, K. H., Nobi, A., Rakib, M. H., & Lee, J. W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04412-y · Cited by 11
“…(NLP) applications, including language modeling, machine translation, sentiment analysis, text classi fi cation, named entity recognition, and text generation (Huang et al. 2019; Ryu et al. 2017; Wen et al. 2019; Y. Wang et al. 2016; Chiu and Nichols, 2016; McCann et al. 2017; Song et al. 2019).”
DEVICE: Depth and Visual Concepts Aware Transformer for OCR-based image captioning
Xu, D., Huang, Q., Zhang, X., Cheng, H., Shuang, F., & Cai, Y. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111522 · Cited by 8
Visual-Language Scene-Relation-Aware Zero-Shot Captioner
Bao, Q., Liu, F., Jiao, L., Liu, Y., Li, S., Li, L., Liu, X., & Chen, P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3581174 · Cited by 13
“These methods were foundational, leveraging sequential modeling to generate captions, but were later improved with the introductionofattentionmechanisms[23],[38],whichenhanced alignmentbetweenimageregionsandtextualdescriptions.”
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
