The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Cited by
Sorted by citations
Items 201–300 of 1,002
Weakly-Supervised 3D Spatial Reasoning for Text-Based Visual Question Answering
Li, H., Huang, J., Jin, P., Song, G., Wu, Q., & Chen, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3276570 · Cited by 40
Self-citationA Review of Transformer-Based Approaches for Image Captioning
Ondeng, O., Absaloms, H. O., & Akuon, P. O. (2023) · Applied Sciences · doi:10.3390/app131911103 · Cited by 39
Prior Knowledge-Guided Transformer for Remote Sensing Image Captioning
Meng, L., Wang, J., Yang, Y., & Xiao, L. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3328181 · Cited by 38
Top-down framework for weakly-supervised grounded image captioning
Cai, C., Wang, S., Yap, K., & Wang, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111433 · Cited by 33
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
“After introducing attention mechanisms, the feature representation of important regions in the current moment’s image serves as a context vector input to the encoding-decoding model [246, 247, 248, 186, 249].”
MethodText Augmentation Using BERT for Image Captioning
Atliha, V., & Šešok, D. (2020) · Applied Sciences · doi:10.3390/app10175978 · Cited by 36
“Additionally, we have confirmed our results on the other state-of-the-art models, such as [41,42], on the best variant of dataset chosen based on the previous experiments with [40].”
ResultInteractions Guided Generative Adversarial Network for unsupervised image captioning
Cao, S., An, G., Zheng, Z., & Ruan, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.08.019 · Cited by 33
BM-NAS: Bilevel Multimodal Neural Architecture Search
Yin, Y., Huang, S., & Zhang, X. D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20872 · Cited by 35
“For instance, AoANet (Huang et al. 2019) proposes the attention on attention (AoA) module, showing that adding an attention operation on top of another one could achieve better performance on image captioning task.”
BackgroundMethodInfluentialA feature consistency driven attention erasing network for fine-grained image retrieval
Zhao, Q., Wang, X., Lyu, S., Liu, B., & Yang, Y. F. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108618 · Cited by 33
“[10, 11, 12, 13] researches on attention-based methods that just exploits image-level annotations to guide the deep network focusing on the most essential region with regard to each category.”
BackgroundOn Distinctive Image Captioning via Comparing and Reweighting
Wang, J., Xu, W., Wang, Q., & Chan, A. B. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3159811 · Cited by 35
“; cNg, which achieve state-of-the-art performance on image captioning tasks [29], [35].”
BackgroundHossain, E., Sharif, O., Hoque, M. M., Dewan, M. A. A., Siddique, N. H., & Hossain, M. A. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.06.010 · Cited by 32
“Multi-modal learning is widely used for various NLP problems, including image captioning (Huang et al., 2019) and visual question answering (Agrawal et al., 2015).”
Multi-Modal Image Captioning for the Visually Impaired
Ahsan, H., Bhatt, D., Shah, K., & Bhalla, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2021.naacl-srw.8 · Cited by 39
“We refer readers to the original work (Huang et al., 2019) for more details.”
MethodInfluentialMemory-Augmented Image Captioning
Fei, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16220 · Cited by 32
“Baselines We equip the proposed memory augmented method with different state-of-the-art methods, including Up-Down (Anderson et al. 2018) and AoANet (Huang et al. 2019).”
BackgroundMethodInfluentialImage Search with Text Feedback by Deep Hierarchical Attention Mutual Information Maximization
Gu, C., Bu, J., Zhang, Z., Yu, Z. H., Ma, D., & Wang, W. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475619 · Cited by 32
“Recently, several self-attention mechanisms are proposed for VQA [17, 32, 34, 70], which builds upon transformer to learn the inter-modal or intra-modal latent attention.”
BackgroundSelf-citationDeep Semantic and Attentive Network for Unsupervised Video Summarization
Zhong, S.-H., Lin, J., Lu, J., Fares, A. H., & Ren, T. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3477538 · Cited by 34
“It has been widely used in various tasks, such as machine translation [35], image caption [12], and visual question answering [21].”
MethodTransforming Visual Scene Graphs to Image Captions
Yang, X., Peng, J., Wang, Z., Xu, H., Ye, Q., Li, C., Huang, S., Huang, F., Li, Z., & Zhang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.694 · Cited by 32
“Based on this new backbone, researchers develop more advanced strategies for further improving the effectiveness, including designing more sophisticated attention mechanisms (Huang et al., 2019; Pan et al., 2020), introducing additional memory blocks (Cornia et al.”
BackgroundMethodInfluentialDeep Learning Image Captioning in Construction Management: A Feasibility Study
Xiao, B., Wang, Y., & Kang, S.-C. (2022) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0002297 · Cited by 36
Visual-Textual Attentive Semantic Consistency for Medical Report Generation
Zhou, Y., Huang, L., Zhou, T., Fu, H., & Shao, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00395 · Cited by 32
“Be different from previous attention modules [43, 18], in this work, the regional visual features are both self-attended and correlated with the hidden states of the sentence decoder to obtain semantic attentions at different topic steps.”
BackgroundMethodSelf-citationDirection Relation Transformer for Image Captioning
Song, Z., Zhou, X., Dong, L., Tan, J., & Guo, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475607 · Cited by 31
“Recently, such an encoder-decoder framework is well promoted with Transformer [28] architectures, where the multi-head attention is efficiently exploited to capture intra- or inter-modality visual relationships [6, 10, 18, 21, 26, 31].”
BackgroundMethodInfluentialA Text-Guided Generation and Refinement Model for Image Captioning
Wang, D., Hu, Z., Zhou, Y., Hong, R., & Wang, M. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3154149 · Cited by 32
“Furthermore, this framework has experienced several significant upgrades, such as attention mechanism [4]–[8] and reinforced mechanism for sequence decoding [9], [10].”
BackgroundMethodResultInfluentialCollaborative Transformers for Grounded Situation Recognition
Cho, J., Yoon, Y., & Kwak, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01904 · Cited by 32
“Visual reasoning such as image captioning [2, 9, 12, 32, 39], scene graph generation [14, 24, 35, 36], and human-object-interaction detection [15,20,33,40] has been widely studied for comprehensive understanding of images.”
BackgroundTang, L., Yi, J., & Li, X. (2023) · Journal of Integrative Agriculture · doi:10.1016/j.jia.2023.06.023 · Cited by 30
With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning
Barraco, M., Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00282 · Cited by 30
“The introduction of Transformer-based models in image captioning has also brought to the development of effective variants of the self-attention operator [10,14,16,19,30,36] and to that of vision-and-language early-fusion architectures [18, 26, 70] based on BERT-like models [11].”
MethodSemantic-Spatial Collaborative Perception Network for Remote Sensing Image Captioning
Wang, Q., Yang, Z., Ni, W., Wu, J., & Li, Q. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3502805 · Cited by 36
“Huang et al. [20] extend this idea by integrating attention mechanisms into Transformer networks to filter out irrelevant visual features.”
Towards Unique and Informative Captioning of Images
Wang, Z., Feng, B. T., Narasimhan, K., & Russakovsky, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58571-6_37 · Cited by 29
“For instance, we find that state-of-the-art captioning systems [2,27,8] incorrectly assign higher likelihoods to irrelevant common captions compared to even ground truth captions paired with a particular image.”
BackgroundMethodInfluentialMulti-Gate Attention Network for Image Captioning
Jiang, W., Li, X., Hu, H., Lu, Q., & Liu, B. (2021) · IEEE Access · doi:10.1109/access.2021.3067607 · Cited by 32
“Recently, with the introduction of transformer [8], some works [4], [10] extend this paradigm by adding an additional transformer-like encoder.”
BackgroundMethodInfluentialEnhance understanding and reasoning ability for image captioning
Wei, J., Li, Z., Zhu, J., & Ma, H. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03624-y · Cited by 30
“[28] introduced a method of using Transformer to encode regions into hidden states, paired with LSTMbased decoders for image captioning.”
MethodInfluentialTang, Y., Yang, H., Zhang, L., & Yuan, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121442 · Cited by 31
A Dual-Feature-Based Adaptive Shared Transformer Network for Image Captioning
Shi, Y., Xia, J., Zhou, M., & Cao, Z. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3353830 · Cited by 31
“The compared models include Up-Down [7], SGAE [48], M 2 Transformer [30], X-Transformer [17], AoA Transformer [29], DLCT [10], APN [49], DSPLTN [11], APST [35], SCD-Net [50], and GIT [16].”
MethodProphet Attention: Predicting Attention with Future Attention
Liu, F., Ren, X., Wu, X., Ge, S., Fan, W., Zou, Y., & Sun, X. (2020) · neural information processing systems · Cited by 30
Connecting What to Say With Where to Look by Modeling Human Attention Traces
Meng, Z., Yu, L., Zhang, N., Berg, T. L., Damavandi, B., Singh, V., & Bearman, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01249 · Cited by 27
“Architectures for both tasks often rely on attention [38, 15, 17, 25], a mechanism inspired by the human visual system [31, 11].”
TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning
Fan, Z., Wei, Z., Wang, S., Wang, R., Li, Z., Shan, H., & Huang, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/91 · Cited by 29
Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools
Lotfi, F., Beheshti, A., Farhood, H., Pooshideh, M., Jamzad, M., & Beigy, H. (2023) · Algorithms · doi:10.3390/a16030135 · Cited by 39
“Therefore, Huang et al. [86] proposed the “At-tention on Attention” (AoA) module, which specifies the attention results and queries relevancy.”
MiniVLM: A Smaller and Faster Vision-Language ModelPreprint
Wang, J., Hu, X., Zhang, P., Li, X., Wang, L., Zhang, L., Gao, J., & Liu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06946 · Cited by 59
“7 - - - AoANet [11] 37.”
Visual Commonsense Representation Learning via Causal Inference
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00197 · Cited by 28
“Feature Model B4 M R C Model B4 M R C Origin [1, 7]”
BackgroundMethodInfluentialPartially Non-Autoregressive Image Captioning
Fei, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16219 · Cited by 28
“Following (Huang et al. 2019), all the sentences are converted to lower case, and we omit words which occur less than five times.”
MethodCAST: Cross-Modal Retrieval and Visual Conditioning for image captioning
Cao, S., An, G., Cen, Y., Yang, Z., & Lin, W. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110555 · Cited by 28
Deconfounded Image Captioning: A Causal RetrospectPreprint
Yang, X., Zhang, H., & Cai, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03923 · Cited by 43
Diverse Image Captioning with Context-Object Split Latent Spaces
Mahajan, S., & Roth, S. (2020) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.2011.00966 · Cited by 27
“Classical approaches for image captioning are framed as sequence modeling task, where image representations from convolutional neural networks (CNNs) are input to recurrent neural networks (RNNs) to generate a single caption for an image [15, 19, 23, 30, 33, 41, 46].”
BackgroundCIDEr-R: Robust Consensus-based Image Description Evaluation
dos Santos, G. O., Colombini, E. L., & Avila, S. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.39 · Cited by 32
“…collection of architectures has been proposed in image cap-tioning (Kulkarni et al., 2011; Vinyals et al., 2015; Johnson et al., 2016; Pedersoli et al., 2017; Anderson et al., 2018; Lu et al., 2018; Huang et al., 2019; Wang et al., 2020; Cornia et al., 2020; Pan et al., 2020; Li et al., 2020).”
Using Neural Encoder-Decoder Models With Continuous Outputs for Remote Sensing Image Captioning
Ramos, R., & Martins, B. (2022) · IEEE Access · doi:10.1109/access.2022.3151874 · Cited by 29
“…are currently being used as the encoder and decoder, respectively, can in the future perhaps also be replaced by new components based on the Transformer architecture, given that these are currently achieving state-of-the-art on a variety of language processing and computer vision tasks [41]–[45].”
MethodMulti-Modal Fake News Detection via Bridging the Gap between Modals
Liu, P., Qian, W., Xu, D., Ren, B., & Cao, J. (2023) · Entropy · doi:10.3390/e25040614 · Cited by 32
“One trend that has emerged in the field is the use of attention mechanisms [36–40], which explored attention to effectively incorporate both global and local visual features in image captioning.”
Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning
Chen, S., Zhu, H., Li, M., Chen, X., Guo, P., Lei, Y., Yu, G., Li, T., & Chen, T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387838 · Cited by 37
Gao, D., Kong, M. C., Zhao, Y., Huang, J.-Z. J., Huang, Z., Kuang, K., Wu, F. Q., & Zhu, Q. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102982 · Cited by 31
Sharma, D., Dhiman, C., & Kumar, D. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3405573 · Cited by 27
“Also, from Table III, it is evident that the our proposed RFCG module provided signi fi cant results when compared with many recent state of the art [20], [25], [58], [59], [60].”
Improving Intra- and Inter-Modality Visual Relation for Image Captioning
Wang, Y., Zhang, W., Liu, Q., Zhang, Z., Gao, X., & Sun, X. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413877 · Cited by 26
“…Network and encodes them into feature vector as additional input; SGAE [36], which introduces auto-encoding scene graphs into their network; AoANet [14], which proposes attention on attention architecture to improve the relevance between query and keys in the vanilla attention mechanism.”
BackgroundIterative Back Modification for Faster Image Captioning
Fei, Z. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413901 · Cited by 26
“(5) AoANet [15] uses extra learned attention on top of self-attention to avoid attentions that are irrelevant to the decoder, which serves as a current state-of-the-art autoregressive captioning model.”
MethodIntegrated generalized zero-shot learning for fine-grained classification
Shermin, T., Teng, S. W., Sohel, F. A., Murshed, M., & Lu, G. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108246 · Cited by 25
“However, some of them do not explore proper guidance from attributes [9, 10] and others ignore local visual details [14, 15].”
BackgroundEfficient Action Recognition via Dynamic Knowledge Propagation
Kim, H., Jain, M., Lee, J.-T., Yun, S., & Porikli, F. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01346 · Cited by 27
“Attention mechanisms have yielded significant improvement in many tasks including image classification [17], object detection [41], image captioning [18], and machine translation [25, 38].”
BackgroundKOHTD: Kazakh offline handwritten text dataset
Toiganbayeva, N., Kasem, M. S., Abdimanap, G. S., Bostanbekov, K. A., Abdallah, A., Alimova, A. N., & Nurseitov, D. B. (2022) · Signal Processing Image Communication · doi:10.1016/j.image.2022.116827 · Cited by 35
“Deep learning has been widely used in several fields nowadays, such as Medical applications like Cancers diagnoses, detection, and classification [1], image captioning [2], speech recognition [3] and in medical question answers [4, 5, 6], table detection and recognition [7, 8] also deep learning has been used in softw…”
MethodMedSkip: Medical Report Generation Using Skip Connections and Integrated Attention
Pahwa, E., Mehta, D., Kapadia, S., Jain, D., & Luthra, A. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00380 · Cited by 27
“More recently, the introduction of the Transformer [23] has led to the use of Transformer-based models for image captioning [9, 11, 4].”
MethodSequential Transformer via an Outside-In Attention for image captioning
Wei, Y., Wu, C., Li, G., & Shi, H. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104574 · Cited by 26
Human-Centric Image Captioning
Yang, Z., Wang, P., Chu, T., & Yang, J. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108545 · Cited by 28
ACORT: A compact object relation transformer for parameter efficient image captioning
Tan, J. H., Tan, Y. H., Chan, C. S., & Chuah, J. H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.081 · Cited by 26
“tion process on salient visual features from the image encoder [18, 19].”
BackgroundAn extended attention mechanism for scene text recognition
Xiao, Z., Nie, Z., Song, C., & Chronopoulos, A. T. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117377 · Cited by 26
Wang, Q., Deng, H., Wu, X. M., Yang, Z., Liu, Y., Wang, Y., & Hao, G. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.010 · Cited by 26
Semantic Representations With Attention Networks for Boosting Image Captioning
Hafeth, D. A., Kollias, S., & Ghafoor, M. (2023) · IEEE Access · doi:10.1109/access.2023.3268744 · Cited by 31
“Figure 4 shows a few images from the MS-COCO dataset with their generated descriptions, the state-of-the-art model descriptions [21], and ground-truth descriptions.”
BackgroundMethodInfluentialEmbedded Heterogeneous Attention Transformer for Cross-Lingual Image Captioning
Song, Z., Hu, Z., Zhou, Y., Zhao, Y., Hong, R., & Wang, M. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3384678 · Cited by 25
“Theexistingcross-lingual imagecaptioningmethods[9],[10],[11],[12],[13],[14],[15], [16],[17]arelimitedandoftenrelyonmultistepimplementa-tions.”
Albadarneh, I., HAMMO, B. H., & Al-Kadi, O. S. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100766 · Cited by 28
“A separate study by [113] improved the attention mechanisms used in image captioning.”
Degradation Aware Approach to Image Restoration Using Knowledge Distillation
Suin, M., Purohit, K., & Rajagopalan, A. N. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3043622 · Cited by 25
“Although an adaptive module should learn to ignore bad-pixels, few recent works like [66] argue about the drawback of standard methods for the task of image captioning and apply additional filtering to remove unnecessary information.”
BackgroundL-Verse: Bidirectional Generation Between Image and Text
Kim, T., Song, G., Lee, S., Kim, S., Seo, Y., Lee, S., Kim, S. H., Lee, H., & Bae, K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01603 · Cited by 25
Expert-defined Keywords Improve Interpretability of Retinal Image Captioning
Wu, T.-W., Huang, J.-H., Lin, J. L., & Worring, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00190 · Cited by 26
“In [24], the authors propose an attention on attention (AoA) module to determine the relevance between attention results and queries.”
BackgroundCross-Modal Text Steganography Against Synonym Substitution-Based Text Attack
Peng, W., Wang, T., Qian, Z., Li, S., & Zhang, X. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3258862 · Cited by 25
“Image Encoder: For the purpose of extracting good semantic feature representation of input image, we apply a general ResNet-101 encoder [25] followed by attention on attention (AOA) mechanism [26].”
MethodArabic Image Captioning using Pre-training of Deep Bidirectional Transformers
Emami, J., Nugues, P., Elnagar, A., & Afyouni, I. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.4 · Cited by 25
“Huang et al. (2019) take the attention concept one step further in their work, where they propose an “Attention on Attention” (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries.”
BackgroundEgocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring
Qiu, J., Lo, F. P.-W., Gu, X., Jobarteh, M. L., Jia, W., Baranowski, T., Steiner‐Asiedu, M., Anderson, A. K., McCrory, M. A., Sazonov, E. E., et al. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3243999 · Cited by 25
“AoANet was introduced in [71], which integrated with a novel attention on attention (AoA) module to better model the relevance between the queries and attention results in attention-based image captioning.”
Unifying Event Detection and Captioning as Sequence Generation via Pre-training
Zhang, Q., Song, Y., & Jin, Q. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_21 · Cited by 24
Wu, X., Li, Y., Long, J., Zhang, S., Wan, S., & Mei, S. (2023) · Remote Sensing · doi:10.3390/rs15020347 · Cited by 29
Relational-Convergent Transformer for image captioning
Chen, L., Yang, Y.-F., Hu, J., Pan, L., & Zhai, H. (2023) · Displays · doi:10.1016/j.displa.2023.102377 · Cited by 24
Text Steganalysis Based on Hierarchical Supervised Learning and Dual Attention Mechanism
Peng, W., Li, S., Qian, Z., & Zhang, X. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3319975 · Cited by 25
“In order to enhance the confidence degree of the attentive semantic and statistical features and avoid misleading information [56], the second point-wise attention is used.”
BackgroundAERMNet: Attention-enhanced relational memory network for medical image report generation
Zeng, X., Liao, T., Xu, L., & Wang, Z. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107979 · Cited by 24
Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning
Li, J., Mao, Z., Li, H., Chen, W., & Zhang, Y. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3638558 · Cited by 24
“An image captioning process involves encoding the visual entities in an image and automatically decoding a descriptive sentence.”
MethodTNT: An Interpretable Tree-Network-Tree Learning Framework using Knowledge Distillation
Li, J., Li, Y., Xiang, X., Xia, S., Dong, S., & Cai, Y. (2020) · Entropy · doi:10.3390/e22111203 · Cited by 23
“Deep Neural Networks (DNNs) have achieved great success in many multimodal prediction tasks such as cross-modal embedding [1], image caption [2], and visual question answering [3].”
BackgroundConfidence-aware Non-repetitive Multimodal Transformers for TextCaps
Wang, Z., Bao, R., Wu, Q., & Liu, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16389 · Cited by 25
“State-of-the-art models for conventional Image Captioning like BUTD (Anderson et al. 2018), AoANet (Huang et al. 2019) fail to describe text in TextCaps images.”
InfluentialA visual persistence model for image captioning
Wang, Y., Xu, J., & Sun, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.10.014 · Cited by 23
Efficient Channel Attention Based Encoder–Decoder Approach for Image Captioning in Hindi
Mishra, S. K., Rai, G., Saha, S., & Bhattacharyya, P. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3483597 · Cited by 27
“This procedure builds a hierarchy which is further used by tree-structured LSTM. Huang et al. [18] have developed attention on attention mechanisms to determine the relevance between attention results and queries.”
MethodI3N: Intra- and Inter-Representation Interaction Network for Change Captioning
Yue, S., Tu, Y., Li, L., Yang, Y., Gao, S., & Yu, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3242142 · Cited by 24
“Thanks to the rapid development of image captioning [14], [15], [16], [17], the existing change captioning methods have made significant inspiration.”
BackgroundCOME: Clip-OCR and Master ObjEct for text image captioning
Lv, G., Sun, Y., Nian, F., Zhu, M., Tang, W., & Hu, Z. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104751 · Cited by 23
Vocabulary-Wide Credit Assignment for Training Image Captioning Models
Liu, H., Zhang, S., Lin, K., Wen, J. A., Li, J., & Hu, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3051476 · Cited by 22
“Following that work, Rennie et al. [14] improved the baseline of REINFORCE and proposed Self-Critical Sequence Training (SCST), which has become the most popular RL algorithm for image captioning nowadays [8], [15], [16], [17].”
BackgroundMethodGlobal-Attention-Based Neural Networks for Vision Language Intelligence
Liu, P., Zhou, Y., Peng, D., & Wu, D. O. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003402 · Cited by 22
“Most of the current approaches employed the encoder-decoder framework, in which encoder usually consists of convolutional neural networks (CNNs), in order to obtain a fixed length of feature vectors from input image as visual representation; while the decoder adopts recurrent neural networks (RNNs) [1], [3]–[6] or att…”
MethodDistributed Attention for Grounded Image Captioning
Chen, N., Pan, X., Chen, R., Yang, L., Lin, Z., Ren, Y., Yuan, H., Guo, X., Huang, F., & Wang, W. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475354 · Cited by 21
“The attention mechanism can be applied to various types of data including spatial grids [30], semantic metadata [49, 50] as well as object region proposals [2, 18].”
BackgroundSelf-citationCaptioning Transformer With Scene Graph Guiding
Chen, H., Wang, Y., Yang, X. S., & Li, J. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506193 · Cited by 22
“2 ) we leverage an attention mechanism [7] to perform cross-modal feature fusion for later caption generation.”
MethodSemi-Autoregressive Image Captioning
Yan, X., Fei, Z., Li, Z., Wang, S., Huang, Q., & Tian, Q. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475179 · Cited by 22
“Most image captioning systems follow an encoder-decoder paradigm [2, 6, 7, 18, 30, 34, 38, 40].”
BackgroundMethodRethinking the Reference-based Distinctive Image Captioning
Mao, Y., Chen, L., Jiang, Z., Zhang, D., Zhang, Z., Shao, J., & Xiao, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548358 · Cited by 22
“Within this framework, many efforts have been made to improve the architecture, including attention mechanisms [1, 5, 7, 14, 46], graph convolution networks [8, 50, 51], and transformer-based models [10, 17].”
BackgroundFashion-Oriented Image Captioning with External Knowledge Retrieval and Fully Attentive Gates
Moratelli, N., Barraco, M., Morelli, D., Cornia, M., Baraldi, L., & Cucchiara, R. (2023) · Sensors · doi:10.3390/s23031286 · Cited by 24
“Some of these works proposed self-attention only in the image encoding stage [27,28], while others introduced instead fully attentive architectures with self-attention mechanisms in both image encoder and language decoder modules [29–34].”
BackgroundMethodCOVID-VTS: Fact Extraction and Verification on Short Video Platforms
Liu, F., Yacoob, Y., & Shrivastava, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.14 · Cited by 33
Image Captioning With Controllable and Adaptive Length Levels
Ding, N., Deng, C., Tan, M., Du, Q., Ge, Z., & Wu, Q. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3328298 · Cited by 23
Xiao, S., Chen, Y., Song, Y., Chen, L., Sun, L., Zhen, Y., Chang, Y., & Zhou, T. (2024) · Displays · doi:10.1016/j.displa.2024.102679 · Cited by 22
Stealthy Targeted Backdoor Attacks Against Image Captioning
Fan, W., Li, H., Jiang, W., Hao, M., Yu, S., & Zhang, X. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3402179 · Cited by 22
“More recently, Huang et al. [20] improved the functionality of attention-based models by proposing the Attention on Attention module to measure the correlation between value and query, while Wang et al. [3] introduced the Multiway Transformer for generic modeling, which yielded deep fusion and…”
MethodVisual Global-Salient-Guided Network for Remote Sensing Image-Text Retrieval
He, Y., Xu, X., Chen, H., Li, J., & Pu, F. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3466389 · Cited by 22
“1) Visual-Guided Module (VGM): Inspired by [45] and [46], we design VGM with the structure shown in Fig.”
MethodCross-Modal Augmented Transformer for Automated Medical Report Generation
Tang, Y., Yuan, Y., Tao, F., & Tang, M. (2025) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2025.3536441 · Cited by 25
“The testing time for each batch of the AOA [5] model is 303 ms.”
Ali, R., Ali, A., Naeem, H. M. Y., Asad, M., Alsarhan, T., & Heyat, M. B. B. (2024) · ICCK Transactions on Advanced Computing and Systems · doi:10.62762/tacs.2024.795448 · Cited by 48
Relation constraint self-attention for image captioning
Ji, J., Wang, M., Zhang, X., Lei, M., & Qu, L. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.062 · Cited by 21
Image captioning using transformer-based double attention network
Parvin, H., Naghsh‐Nilchi, A. R., & Mohammadi, H. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106545 · Cited by 21
A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes
Yu, T., Lin, X., Wang, S., Sheng, W., Huang, Q., & Yu, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3296889 · Cited by 24
“Furthermore, some approaches [49], [57], [58], [59], [60] incorporated object detectors to enhance the extraction of visual features.”
BackgroundHou, X., Li, X., Liu, Z., Sang, S., Lu, M., & Zhang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126394 · Cited by 20
Abdulgalil, H. D., & Basir, O. A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100159 · Cited by 20
Topic Scene Graph Generation by Attention Distillation from Caption
Wang, W., Wang, R., & Chen, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01560 · Cited by 20
“Modern captioning models have achieved great progress benefiting from encoder-decoder framework [49], attention technique [4, 7, 15, 17, 24, 37, 48, 56, 60] and RL-based training objective [41].”
BackgroundSelf-citationGroup-based Distinctive Image Captioning with Memory Attention
Wang, J., Xu, W., Wang, Q., & Chan, A. B. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475215 · Cited by 20
“We compare our model with two groups of state-of-the-art models: 1) FC [24], Att2in [24], UpDown [1] and AoANet [12] that aim to generate captions with high accuracy; 2) DiscCap [18], CL-Cap [8], and CIDErBtwCap [31] that generate distinctive captions.”
BackgroundMethodInfluentialAn Object Localization-based Dense Image Captioning Framework in Hindi
Mishra, S. K., Harshit, Saha, S., & Bhattacharyya, P. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3558391 · Cited by 21
“Attention on attention mechanism is introduced in [19].”
BackgroundModeling graph-structured contexts for image captioning
Li, Z., Wei, J., Huang, F., & Ma, H. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104591 · Cited by 20
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
