The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Cited by
Sorted by citations
Items 501–600 of 1,002
Automatic Chinese Meme Generation Using Deep Neural Networks
Wang, L., Zhang, Q., Kim, Y., Wu, R., Jin, H., Deng, H., Luo, P., & Kim, C. (2021) · IEEE Access · doi:10.1109/access.2021.3127324 · Cited by 9
Incorporating retrieval-based method for feature enhanced image captioning
Zhao, S., Li, L., & Peng, H. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04010-4 · Cited by 8
“Since AoA and MMT mainly focus on the learning of visual features and their relationships, without induce of the extra informative features, which is useful for deeply understand the image.”
BackgroundResultTang, Y., Zhang, L., Yuan, Y., & Chen, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3233369 · Cited by 8
“We compare the proposed LSAM-ARS with several classic approaches including AdaAtt [13], Att2In [43], UpDown [16], AOA [15], Mesh-Memory Transformer [19] and SRFC [10].”
MethodSpatial-aware topic-driven-based image Chinese caption for disaster news
Zhou, J., Zhu, Y., Zhang, Y., Yang, C., & Pan, H. (2023) · Neural Computing and Applications · doi:10.1007/s00521-022-08072-w · Cited by 8
“Attention-based methods [2 – 4 , 56] add a visual attention mechanism to the encoder / decoder framework to enhance the utilization of In consideration of the image feature representation that will affect the dependence of the generated sentence on the images, numerous studies focus on the image…”
MethodA Review of Stylized Image Captioning Techniques, Evaluation Parameters, and Datasets
Sharma, D., Dhiman, C., & Kumar, D. (2022) · International Conference on Artificial Intelligence and Speech Technology (AIST) · doi:10.1109/aist55798.2022.10064842 · Cited by 8
“In recent years, caption generation models [4] [5] [6] has achieved appreciable success however, these models focus only on generation of factual description of images but ignores the linguistic styles of the sentences.”
BackgroundEmbodied Agents for Efficient Exploration and Smart Scene Description
Bigazzi, R., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160668 · Cited by 8
“The image representation is then used to condition a recurrent neural network [24], [30], [31] or a Transformer-based [32], [33] language model that generates the caption.”
MethodKalarani, A. R., Bhattacharyya, P., Chhaya, N., & Shekhar, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.67 · Cited by 8
CDKM: Common and Distinct Knowledge Mining Network With Content Interaction for Dense Captioning
Deng, H., Xie, Y., Wang, Q., Wang, J., Ruan, W., Liu, W., & Liu, Y. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3407695 · Cited by 8
“[10]extendthetraditional attentionmechanismintheencoderanddecoder,thusdetermin-ingtherelevancebetweentheattentionresultandthequery.”
Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training
Sarto, S., Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02535-y · Cited by 18
“…and RNNs (Karpa-thy & Fei-Fei, 2015; Vinyals, Toshev, Bengio, & Erhan, 2015) to the latest fully attentive architectures (Cornia et al., 2020; Huang et al., 2019; Y. Li et al., 2022; Pan et al., 2020), generate cap-tions by greedily selecting the most probable word from a learned vocabulary.”
iCap: Interactive Image Captioning with Predictive Text
Jia, Z., & Li, X. (2020) · International Conference on Multimedia Retrieval · doi:10.1145/3372278.3390697 · Cited by 8
“Novel visual captioning models are being actively developed [5, 8, 10, 13] with increasing performance reported on public benchmark datasets such as MS-COCO [6] and MSR-VTT [24].”
BackgroundMethodInfluentialAIMH at SemEval-2021 Task 6: Multimodal Classification Using an Ensemble of Transformer Models
Messina, N., Falchi, F., Gennaro, C., & Amato, G. M. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.140 · Cited by 9
“, 2017), image captioning (Zhou et al., 2020; Rennie et al., 2017; Huang et al., 2019; Cornia et al., 2019), and image-text matching (Chen et al.”
BackgroundDiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention
Liu, F., Wu, X., Ge, S., Ren, X., Fan, W., Sun, X., & Zou, Y. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3447685 · Cited by 9
“2 AoANet [27] - - - - - - - - - - - - - - - 38.”
MethodInfluentialDeep Metric Network Via Heterogeneous Semantics for Image Sentiment Analysis
Liang, Y., Maeda, K., Ogawa, T., & Haseyama, M. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506701 · Cited by 8
“Therefore, the image captioning [12–14], which is a more exhaustive factor consisted of the descriptions, can be expected to be effective for image sentiment analysis.”
BackgroundPartial Off-policy Learning: Balance Accuracy and Diversity for Human-Oriented Image Captioning
Shi, J., Li, Y., & Wang, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00219 · Cited by 8
“We apply the partial off-policy learning scheme on [14, 34, 13] in Fig.”
BackgroundMethodInfluentialSynthesizing Spoken Descriptions of Images
Wang, X., van der Hout, J., Zhu, J., Hasegawa‐Johnson, M., & Scharenborg, O. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3120644 · Cited by 8
“B. Image captioning methods Several image captioning models [1], [4], [20], [25], [9], including a basic encoder-decoder architecture (Neural Image Caption [20]), a standard attention-guided model (Show, Attend and Tell [1]), and several state-of-the-art models (Updown model [25], Attention on Attention Model [4], and…”
BackgroundMethodInfluentialVision Transformer Based Model for Describing a Set of Images as a Story
Malakan, Z. M., Hassan, G. M., & Mian, A. S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22695-3_2 · Cited by 8
“Approaches may be further classified as rule-based methods [2,23] and deep learning-based methods [27,11].”
MethodWang, H., Niu, J., Liu, X., & Wang, Y. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995003 · Cited by 8
“In recent years, there have emerged several image captioning methods [9]–[11], [16], [17] based on the encoder-decoder framework, which generate a short descriptive sentence given an image.”
BackgroundAdvances in Convolutional Neural Networks
Wen, X., He, J., Shu, Y., & Zheng, H. (2020) · IntechOpen eBooks · doi:10.5772/intechopen.93512 · Cited by 11
Understanding Image Captioning Models beyond Visualizing AttentionPreprint
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2020) · arXiv (Cornell University) · Cited by 8
Doore, S. A., Istrati, D., Xu, C., Qiu, Y., Sarrazin, A., & Giudice, N. A. (2024) · Journal of Imaging · doi:10.3390/jimaging10010026 · Cited by 11
“Transformer-based approaches [42] have been applied to image captioning tasks using complex multiple layers and fully attentive encoder/decoder mechanisms that connect the image elements to the language model in combination with techniques for processing semantic knowledge [43–45].”
MethodImage captioning: Semantic selection unit with stacked residual attention
Song, L., Li, F., Wang, Y., Liu, Y., Wang, Y., & Xiang, S. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.104965 · Cited by 7
Distilled Cross-Combination Transformer for Image Captioning with Dual Refined Visual Features
Hu, J., & Li, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681161 · Cited by 7
GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths
Chen, X., Jiang, M. X., & Zhao, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_18 · Cited by 11
Enriched Image Captioning Based on Knowledge Divergence and Focus
Liu, A.-A., Wu, Q., Xu, N., Tian, H., & Wang, L. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3525158 · Cited by 10
“Nevertheless, most existing methods [17], [28] ignore the “divergence-focus” nature of the task, as shown in Fig.”
Dynamic window sampling strategy for image captioning
Li, Z., Wei, J., Xian, T., Zhang, C., & Ma, H. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110358 · Cited by 7
Geometric understanding in spatial contexts for image captioning
Hossain, M. A., Hossen, M. B., Ye, Z., Islam, M. S., & Abdullah, M. I. (2026) · Signal Processing Image Communication · doi:10.1016/j.image.2026.117645 · Cited by 8
Cross modification attention-based deliberation model for image captioning
Lian, Z., Zhang, Y., Li, H., Wang, R., & Hu, X. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03845-1 · Cited by 7
“At present, the majority of image captioning models follow the conventional encoder-decoder framework [2]–[6], [11]– [14], where a Convolutional Neural Network (CNN) first”
BackgroundMethodInfluentialA comprehensive survey on deep-learning-based visual captioning
Xin, B., Xu, N., Zhai, Y., Zhang, T., Lu, Z., Liu, J., Nie, W., Li, X., & Liu, A.-A. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01175-x · Cited by 8
“Transformer-based sequence captioners have achieved the state-of-the-art performances recently [66, 86, 157].”
BackgroundInfluentialAn Image Captioning Model Based on SE-ResNest and EMSA
Yuan, R., & Li, H.-S. (2023) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai59366.2023.10332008 · Cited by 7
“In machine translation, the proposed transformer model [6] can well capture the contextual dependencies in natural language and then apply it to the attention-to-attention (AOA) model [18] for image captioning.”
MethodWeakly supervised grounded image captioning with semantic matching
Du, S., Zhu, H., Lin, G., Liu, Y., Wang, D. Y., Shi, J., & Wu, Z. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05389-y · Cited by 7
“Consequently, Huang et al. [21] integrated the attention block into the encoder and decoder to assess the relevance between the attention result and the query.”
Simple Token-Level Confidence Improves Caption Correctness
Petryk, S., Whitehead, S., Gonzalez, J. E., Darrell, T., Rohrbach, A., & Rohrbach, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00564 · Cited by 8
“Image captioning has seen significant progress since the arrival of deep learning as a dominant methodology [5, 14, 26, 27, 44, 57].”
BackgroundIMCN: Improved modular co-attention networks for visual question answering
Liu, C., Wang, C., & Peng, Y. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05456-4 · Cited by 7
“Therefore, we leverage the AoA proposed in [12] to improve thetraditionalmulti-headattentionmodule,whichmeansthat another attention function is used after obtaining the result of multi-head attention.”
Choudhury, P., Guha, P., & Nandi, S. (2024) · International Journal of Asian Language Processing · doi:10.1142/s2717554524500048 · Cited by 7
Self-supervised modal optimization transformer for image captioning
Wang, Y., Li, D., Liu, Q., Liu, L., & Wang, G. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10211-4 · Cited by 7
“In order to make a fair comparison with our models, we select models with pre-trained models to extract features, and the compared models are as follows: SGAE [19], AoANet [57], and X-Transformer [58] all use ResNet101 [59] and Faster R-CNN [14] to extract image grid and region features.”
Improving generative adversarial network inversion via fine-tuning GAN encoders
Yu, C., Wang, W., & Bugiolacchi, R. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112201 · Cited by 9
Self-citationCSDNet: cross-sketch with dual gated attention for fine-grained image captioning network
Hossain, M. S., Aktar, S., Hossen, M. B., Hossain, M. A., Gu, N., & Huang, Z. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20220-z · Cited by 8
“Later innovations,suchasthelook-backmethod[18]andattentiononattentionmodule[27],further refine visual attention by measuring the relevance between attention results and queries.”
A Survey on Automatic Image Captioning Approaches: Contemporary Trends and Future Perspectives
Salgotra, G., Abrol, P., & Selwal, A. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10190-8 · Cited by 9
“Huang et al. [67] developed a new attention-based architecture that includes an additional level of attention.”
Answer-checking in Context: A Multi-modal Fully Attention Network for Visual Question Answering
Huang, H., Han, T., Han, W., Yap, D., & Chiang, C.-M. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413078 · Cited by 7
“Inspired by attention on attention [12] and bilinear attention networks [13], we design a bilinear self-gated input gate to remove irrelevant features for the multi-modal attention module.”
MethodPretrained Word Embeddings for Image Captioning
Atliha, V., & Šešok, D. (2021) · IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream) · doi:10.1109/estream53087.2021.9431465 · Cited by 7
“As a rule, all modern architectures are an encoder-decoder type [6], [7], [8], [9] [10], [11], [12] where a deep convolutional neural network is used as an encoder [13] (usually pretrained and only slightly fine-tuned in the process of training the entire model) and as a decoder an architecture for…”
BackgroundDependent Multi-Task Learning with Causal Intervention for Image Captioning
Chen, W., Tian, J., Fan, C., He, H., & Jin, Y. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/312 · Cited by 8
“Follow-up researches further captured the relationships among objects by adding self-attention on object-level features [Huang et al. , 2019], or using graph convolutional networks [Yao et al. , 2018; Yang et al. , 2019].”
Ogawa, N., Maeda, K., Ogawa, T., & Haseyama, M. (2022) · Sensors · doi:10.3390/s22010382 · Cited by 7
“For example, the literature [16] has reported the effectiveness of using only useful attention by considering the relationship between attention and query in image captioning.”
BackgroundFineFormer: Fine-Grained Adaptive Object Transformer for Image Captioning
Wang, B., Zhang, Z., Fan, J., Zhao, M., Zhan, C., & Xu, M. (2022) · Proceedings · doi:10.1109/icdm54844.2022.00061 · Cited by 7
“The compared methods include LSTM [3], SCST [42], LSTM-A [43], RFNet [44], Up-Down [12], GCN-LSTM [9], ORT [21], AoANet [10], X-Transformer [32].”
MethodCapOnImage: Context-driven Dense-Captioning on Image
Gao, Y., Hou, X., Zhang, Y., Ge, T., Jiang, Y., & Wang, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.226 · Cited by 11
Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
Bucciarelli, D., Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92089-9_22 · Cited by 17
Qin, M. (2022) · Connection Science · doi:10.1080/09540091.2022.2088699 · Cited by 7
“The Attention mechanism (Huang et al., 2019) is mainly proposed to solve a series of problems such as text information ambiguity in the decoding process.”
BackgroundImage captioning with residual swin transformer and Actor-Critic
Zhou, Z., Yang, Y., Li, Z., Zhang, X., & Huang, F. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07848-4 · Cited by 7
“Huang et al. [35] heightened the ability of conventional attention to obtain a more accurate result.”
Reconciling Image Captioning and User’s Comments for Urban Tourism
Bounab, Y., Oussalah, M., & Ferdenache, A. (2020) · International Conference on Image Processing Theory, Tools and Applications (IPTA) · doi:10.1109/ipta50016.2020.9286602 · Cited by 7
“Deep learning-based image captioning methods can use either simple Encoder-Decoder architecture or compositional architecture [14], [15].”
MethodFast Sequence Generation with Multi-Agent Reinforcement LearningPreprint
Guo, L., Liu, J., Zhu, X., & Lu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09698 · Cited by 7
PCATNet: Position-Class Awareness Transformer for Image Captioning
Tang, Z., Yi, Y., Yu, C., & Yin, A. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.037861 · Cited by 8
Rahman, M. M., Uzzaman, A., Sami, S. I., Khatun, F., & Bhuiyan, M. A.-A. (2024) · IET Image Processing · doi:10.1049/ipr2.13287 · Cited by 7
Do as I Can, Not as I Get: Topology-Aware Multi-Hop Reasoning on Multi-Modal Knowledge Graphs
Zheng, S., Yin, H., Chen, T., Nguyen, Q. V. H., Chen, W., & Zhao, L. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3546686 · Cited by 6
“The attended features ˆ V obtained by the attention-fusion module may contain irrelevant features [14].”
BackgroundA Deep Learning Approach for Automated Image Captioning with CNN and LSTM Models
C, S. S. K., Natarajan, S. K., & Ambika, I. (2025) · International Conference on Computational Robotics, Testing and Engineering Evaluation (ICCRTEE) · doi:10.1109/iccrtee64519.2025.11053022 · Cited by 6
Fine-Grained Image Captioning by Ranking Diffusion Transformer
Wan, J., Gan, M., Zhang, L., Zhou, J., Liu, J., Du, B., & Chen, C. L. P. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3641303 · Cited by 6
“Overall, RDT achieves superior performance over state-of-the-art region-based [10], [26], [45], [46] and grid-based models [11], [13], [14].”
Joint Underwater Image Enhancement and Captioning Through Multisupervised Chained Task Learning
Li, H., Li, L., Wang, H., Zhang, W., Liu, J., & Ren, P. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3710783 · Cited by 10
“The introduction of soft attention enabled the model to dynamically focus on salient regions, and methods such as BUTD [34] and AoANet [35] further improved region-level reasoning by incorporating bottom-up cues and more refined attention designs.”
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning
Zhang, H., Sugawara, S., Aizawa, A., Zhou, L., Sasano, R., & Takeda, K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.516 · Cited by 6
Mitigating Dataset Bias in Image Captioning Through Clip Confounder-Free Captioning Network
Kim, Y., Kim, J., Lee, B., Shin, S., & Ro, Y. M. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222502 · Cited by 6
“In Table 1, we compare our C 2 Cap with the SOTA models on the offline MSCOCO Karpathy test split, including SCST [11], Up-Down [20], RFNet [22], GCN-LSTM [23], SGAE [24], ORT [25], AoANet [26], M 2 Transformer [27].”
MethodVisual contextual relationship augmented transformer for image captioning
Su, Q., Hu, J., & Li, Z. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05416-y · Cited by 6
“Therefore, we refer to the AOA [14] and focal loss [15] meth-ods and apply the attention purification method to our image captioning model.”
Shahzadi, I., Madni, T. M., Janjua, U. I., Batool, G., Naz, B., & Ali, M. Q. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01126-6 · Cited by 7
“The primary purpose of image captioning is to provide a short descriptive sentence for a given input image, which has captured extensive research interests [8, 33].”
Huang, X., Lu, K., Wang, S., Lu, J., Li, X., & Zhang, R. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103939 · Cited by 9
Exploring coherence from heterogeneous representations for OCR image captioning
Zhang, Y., Song, Z., & Hu, Z. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01470-1 · Cited by 6
“We compare the performance of our method with conventional image captioning models (Up-Down [10] and AoANet [11]) and text-based image captioning models (M4C-Captioner [1], LSTM-R [14], CNMT [23], OMOT [56] and COME [41]).”
Zhao, K., Lai, R., Guo, B., Liu, L., He, L., & Zhao, Y. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3678537 · Cited by 8
“In addition, research works have also begun to use Transformer and its self-attention mechanism to improve the accuracy as well as richness of description [26, 28, 56].”
Image to Text Conversion Using Deep Learning Algorithms: Survey
D, S. M., Anusuya, V. V., Santhiya, R. V. A., & Srimathi, R. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10716956 · Cited by 6
“Introduced Attention of Attention(AOA) an extension of Convolutional attention [6].”
BackgroundBRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues
Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73229-4_5 · Cited by 7
Perturb, Predict & Paraphrase: Semi-Supervised Learning using Noisy Student for Image Captioning
Jain, A., Samala, P. R., Jyothi, P., Mittal, D., & Singh, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/105 · Cited by 7
An Attentive Fourier-Augmented Image-Captioning Transformer
Raymond, O. I., Yang, Z., & Long, J. (2021) · Applied Sciences · doi:10.3390/app11188354 · Cited by 6
“In imagecaptioning, transformers, with their attention-only architectures (without recurrent units), may follow the encoder–decoder architecture [10,13], wherein the encoder receives and processes image features, and sends them to the decoder module (language model) in which captions are also input and output.”
BackgroundMethodDeep Learning Based Video Captioning in Bengali
Raj, A. H., Seum, A., Dash, A., Islam, S., & Shah, F. M. (2021) · International Conference on Automation and Computing (ICAC) · doi:10.23919/icac50006.2021.9594154 · Cited by 6
Visual question answering with gated relation‐aware auxiliary
Shao, X., Xiang, Z., & Li, Y. (2022) · IET Image Processing · doi:10.1049/ipr2.12421 · Cited by 6
EDDIE-Transformer: Enriched Disease Embedding Transformer for X-Ray Report Generation
Nguyen, H. T., Nie, D., Badamdorj, T., Liu, Y., Hong, L., Truong, J., & Cheng, L. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761459 · Cited by 6
“There exists many different works that build on the work of image captioning [2–4] and adapt it to the specifics of medical report generation [1, 5–18].”
BackgroundBahdanau Attention Based Bengali Image Caption Generation
Alam, M. D. S., Rahman, M. D. S., Hosen, M. D. I., Mubin, K. A., Hossen, S., & Mridha, M. F. (2022) · International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa54658.2022.9765268 · Cited by 7
“In recent time, transformer architecture is adopted by image captioning models [8] [10] [7].”
MethodLeveraging explainability for understanding object descriptions in ambiguous 3D environments
Doğan, F. I., Melsión, G. I., & Leite, I. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.937772 · Cited by 11
A Request for Clarity over the End of Sequence Token in the Self-Critical Sequence Training
Hu, J. C., Cavicchioli, R., & Capotondi, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43148-7_4 · Cited by 6
SynthCap: Augmenting Transformers with Synthetic Data for Image Captioning
Caffagni, D., Barraco, M., Cornia, M., Baraldi, L., & Cucchiara, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43148-7_10 · Cited by 6
“These models are commonly used both in the visual encoding stage [12,21,29,53] and as language models [14,17,30,60], also leading to the design of effective variants of the self-attention operator [14,21,33].”
MethodGenerative adversarial network for semi-supervised image captioning
Liang, 徐. 良. X., Li, C., & Tian, L. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104199 · Cited by 6
VSAM-Based Visual Keyword Generation for Image Caption
Zhang, S., Zhang, Y., Chen, Z., & Li, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3058425 · Cited by 6
“Huang et al. [34] extended the region-level attention mechanism to determine the relevance between attention results and queries, so as to know whether or how well the attended vector and the given attention query are related.”
BackgroundAttention-based networks for analyzing inappropriate speech in Arabic text
Berrimi, M., Moussaouı, A., Oussalah, M., & Saidi, M. (2020) · International Symposium Informatics and its Applications · doi:10.1109/isia51297.2020.9416539 · Cited by 6
“• Attention is the corp mechanism in many state-of-the-art SOTA models like BERT [36], Transformer [37] and used in different tasks such as image captioning [31].”
MethodRevisiting Knowledge Distillation for Image Captioning
Dong, J., Hu, Z., & Zhou, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-93046-2_52 · Cited by 6
Visual-Text Reference Pretraining Model for Image Captioning
Li, P., Zhang, M., Lin, P., Wan, J., & Jiang, M. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9400999 · Cited by 6
“[24], UP-DOWN [45], CAVP [46], SGAE [26], and ASG [27]; (2) pretraining models are used. methods are based on multilayer transformers, such as ORT [47], AOANET [48], NG-SAN [29], and U-VLP [7].”
MethodCNN-LSTM based Social Media Post Caption Generator
Singh, A., & Vij, D. (2022) · International Conference on Innovative Practices in Technology and Management (ICIPTM) · doi:10.1109/iciptm54933.2022.9754189 · Cited by 6
Adaptive Text Denoising Network for Image Caption Editing
Yuan, M., Bao, B., Tan, Z., & Xu, C. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3532627 · Cited by 6
“In addition, the structure with a transformer as decoder [14] has also been well applied in the field of image caption in recent years.”
MethodInfluentialGenerating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets
Cornia, M., Baraldi, L., Fiameni, G., & Cucchiara, R. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01949-w · Cited by 23
“Table 10 shows a comparison when using Up-Down (Anderson et al., 2018) and AoANet (Huang et al., 2019) in a zero-shot manner trained on COCO and when fine-tuning them on the aforementioned datasets.”
InfluentialTowards annotation-free evaluation of cross-lingual image captioning
Chen, A., Huang, X., Lin, H., & Li, X. (2021) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3444685.3446322 · Cited by 6
“upervised manner, with thecross-entropylossminimizedbytheAdamoptimizer.The initial learning rate of Show-Tell and Up-Down is set to be 0.0005. All hyper-parameters of AoANet follow the original paper [9]. The maximum number of training epochs is 80. Best models are selected based on their CIDEr scores on the validatio…”
BackgroundMethodInfluentialB-SCST: Bayesian Self-Critical Sequence Training for Image CaptioningPreprint
Bujimalla, S., Subedar, M., & Tickoo, O. (2020) · arXiv.org · doi:10.48550/arxiv.2004.02435 · Cited by 10
“We presented our results on AoANet architecture, and demonstrated improved CIDEr-D scores on Flickr30k and MS COCO datasets, as compared to SCST approach.”
InfluentialFu, P., Xiao, G., & Yang, H. (2024) · The Visual Computer · doi:10.1007/s00371-024-03372-9 · Cited by 5
“The Transformer is gradually replacing RNN in natural language processing [22, 23] and gaining attention in computer vision [11, 24–30].”
Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning
Luo, J., Chen, J., Li, Y., Pan, Y., Feng, J., Chao, H., & Yao, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_14 · Cited by 5
“More advanced methods are proposed [3,13,18–20,25,26,28,46] to better capture the correlation between visual and linguistic elements.”
Self-citationDeep Reciprocal Learning for Image Captioning
Zhang, J., Zhang, K., Xie, Y., & Wang, Z. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3539344 · Cited by 5
Fusing grid and adaptive region features for image captioning
Wei, J., Li, Z., Zhang, C., & Ma, H. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105513 · Cited by 5
Key region Semantic information Augmented Transformer for Image Captioning
Deng, F., Li, W., & Li, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112135 · Cited by 5
Berger, U., Stanovsky, G., Abend, O., & Frermann, L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.52 · Cited by 13
Mali, S. D., & Agilandeeswari, L. (2024) · Egyptian Informatics Journal · doi:10.1016/j.eij.2024.100514 · Cited by 5
Modality-specific adaptive scaling and attention network for cross-modal retrieval
Ke, X., Chen, B., Cai, Y., Liu, H., Guo, W., & Chen, W. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128664 · Cited by 5
Yang, L., He, L., Hu, D., Liu, Y., Peng, Y., Chen, H., & Zhou, M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3440872 · Cited by 5
“Huang et al. [17], Cornia et al. [18], Ji et al. [19], and Yang et al. [26] introduce several direct or related improvements to the basic transformer model, while their purposes are similar, to intensify the ability of information filtration and multimodal capacity.”
Thangka image captioning model with Salient Attention and Local Interaction Aggregator
Hu, W., Zhang, F., & Zhao, Y. (2024) · Heritage Science · doi:10.1186/s40494-024-01518-5 · Cited by 5
“The AoANet model proposed by Huang et al. [11] combined self-attention with contextual information, enhancing the accuracy of text generation by the decoder through the creation of “information vectors” and “attention gates.”
Zhang, D., Zhang, T., Sun, H., Tang, Y., & Liu, Q. (2024) · Sensors · doi:10.3390/s24237549 · Cited by 5
Graph-based image captioning with semantic and spatial features
Parseh, M. J., & Ghadiri, S. (2025) · Signal Processing Image Communication · doi:10.1016/j.image.2025.117273 · Cited by 7
Self-Attention Meta-Learner for Continual Learning
Sokar, G., Mocanu, D. C., & Pechenizkiy, M. (2021) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/scpy1898 · Cited by 6
Enhanced-Memory Transformer for Coherent Paragraph Video Captioning
Cardoso, L. V., Guimarães, S. J. F., & Patrocínio, Z. K. G. (2021) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai52525.2021.00133 · Cited by 5
Guiding Attention using Partial-Order Relationships for Image Captioning
Popattia, M., Rafi, M., Qureshi, R., & Nawaz, S. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00513 · Cited by 5
“Moroever, the use of self-attention has been shown to improve performance over traditional attention mechanisms such as [11], more notably in transformers [5, 24] and hence can be incorporated with the use of these multi-modal embeddings to improve performance.”
BackgroundMethodInfluentialImage Caption Method Based on Graph Attention Network with Global Context
Sui, J., Yu, H., Liang, X., & Ping, P. (2022) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc55077.2022.9886239 · Cited by 5
“AoA [9] extends the self-attention module to design image caption model.”
BackgroundExplainability in image captioning based on the latent space
Elguendouze, S., Hafiane, A., de Souto, M. C. P., & Lefeuvre-Halftermeyer, A. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126319 · Cited by 5
A Survey on Learning Objects’ Relationship for Image Captioning
Du, R., Zhang, W., Guo, Z. M., & Sun, X. (2023) · Computational Intelligence and Neuroscience · doi:10.1155/2023/8600853 · Cited by 5
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
