The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Cited by
Sorted by citations
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,791
“2 AoANet [11] 37.”
ResultMeshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259
“Noticeably, when using four models our approach achieves the best performance according to all metrics, with an increase of 2.5 CIDEr points with respect to the current state of the art [14].”
BackgroundMethodInfluentialMultimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
“graph [228], video to caption [229], [230], [231], [232], image to caption [233], [234], [235], [236], [237], text to speech [238], text to image [205], [239], text to shape [240], RGB to 3D human”
BackgroundThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Self-citationThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“To help the agent understand the information contained within images, a straightforward approach is to generate corresponding text descriptions for image inputs, known as image captioning [203–207].”
Self-citationVinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 1,015
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
X-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720
“Later on, the most recently proposed attention on attention module [12] enhances visual attention by further measuring the relevance between the attention result and the query.”
BackgroundConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 701
“State-of-the-art V+L models heavily rely on self-attention mechanisms [73] or similar com-5 ponents that allow free information flow among visual regions, and image captioning is no exception [66, 80, 19, 32, 23].”
BackgroundTip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification
Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_29 · Cited by 510
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 701
AiATrack: Attention in Attention for Transformer Visual Tracking
Gao, S., Zhou, C., Ma, C., Wang, X., & Yuan, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_9 · Cited by 414
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring
Suin, M., Purohit, K., & Rajagopalan, A. N. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00366 · Cited by 323
“(6) can be intuitively compared to [4], where similar gated-enhancement technique is used to refine the result by elementwise-multiplication with an attention mask that helps in propagating only the relevant information.”
MethodDual-level Collaborative Transformer for Image Captioning
Luo, Y., Ji, J., Sun, X., Cao, L., Wu, Y., Huang, F., Lin, C.-W., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16328 · Cited by 343
“To explain, most recent methods in image captioning (Huang et al. 2019; Cornia et al. 2020; Pan et al. 2020) use the self-attention modules to model the relationships of visual features.”
BackgroundMethodInfluentialFrom Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373
“Some works adopted the self-attention operator in place of the additive attention one in LSTM-based language models [79], [80], [83], [107].”
BackgroundRSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words
Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01521 · Cited by 303
“[16], GCN-LSTM [43], SGAE [42], ORT [12], AoANet [13], M(2) Transformer [6], and X-Transformer [28].”
BackgroundInfluentialWang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291
“AoANet [25] proposed an “Attention on Attention” module on feature encoder and caption decoder for refining with the self-attention mechanism.”
MethodFine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention
Huynh, D., & Elhamifar, E. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00454 · Cited by 267
“Attention can also be recursively learned in a hierarchical fashion [39, 40].”
BackgroundNormalized and Geometry-Aware Self-Attention Network for Image Captioning
Guo, L., Liu, J., Zhu, X., Yao, P., Lu, S., & Lu, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01034 · Cited by 261
“We compare NGSAN with the state-of-the-art methods, including Up-Down [2], CAVP [23], SGAE [42], VSUA [14], ORT [15], AoANet [17], and MT [43].”
MethodUnifying Vision-and-Language Tasks via Text GenerationPreprint
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 291
“XGPT uses AoA module (Huang et al., 2019) as visual encoder.”
MethodTextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_44 · Cited by 283
“9 3 AoANet [18] COCO 18.”
BackgroundMethodInfluentialExploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
Yuan, Z., Zhang, W., Fu, K., Li, X., Deng, C., Wang, H., & Sun, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3078451 · Cited by 245
“2) Fusion Attention: The motivation comes from attention on attention (AoA) [43].”
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 382
Radiology report generation with a learned knowledge base and multi-modal alignment
Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S. K., & Xiao, L. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102798 · Cited by 199
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 219
“Huang et al. [ 59 ] have introduced a new attention-based structure containing one more level of attention.”
BackgroundMethodInfluentialVisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01750 · Cited by 222
“• AoA Transformer, which inserts an attention-on-attention (AoA) module [24] into every transformer layer, as depicted by Figure 3 (b).”
BackgroundMethodInfluentialSelf-citationJi, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16258 · Cited by 211
“…(Vaswani et al. 2017), where the self-attention is efficiently utilized to capture the correlations among the regions and words (Liu et al. 2019; Huang et al. 2019; Li et al. 2019a; Herdade et al. 2019; Cornia et al. 2020). is region-biased, which only focuses on the region-level information…”
BackgroundMethodInfluentialImage Search With Text Feedback by Visiolinguistic Attention Learning
Chen, Y., Gong, S., & Bazzani, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00307 · Cited by 200
“Recently, several concurrent works [59, 56, 35] extend the pre-training strategies from BERT [9] to learn the latent compositional representations, which jointly represent images and descriptive texts for solving VQA, captioning, or image-text matching.”
BackgroundMethodInfluentialCaptioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182
“For instance, we observe better performance when UpDown [8] and AoANet [30] are trained from scratch on VizWiz-Captions rather than fine-tuned from models pretrained on MSCOCO-Captions, and vice versa for SGAE [56].”
MethodInfluentialCausal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175
TransZero: Attribute-Guided Transformer for Zero-Shot Learning
Chen, S., Hong, Z., Liu, Y., Xie, G.-S., Sun, B., Li, H., Peng, Q., Lü, K., & You, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19909 · Cited by 175
“Some methods (Gabeur et al. 2020; Cornia et al. 2020; Huang et al. 2019; Pan et al. 2020) have also shown that the transformer architecture can better capture the relationship between visual features and process sequences in parallel during training.”
MethodAsymmetric Feature Fusion Network for Hyperspectral and SAR Image Classification
Li, W., Gao, Y., Zhang, M., Tao, R., & Du, Q. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3149394 · Cited by 170
End-to-End Transformer Based Model for Image Captioning
Wang, Y., Xu, J., & Sun, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20160 · Cited by 183
Transformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 194
“Machine translation [101, 102], text summarization [103] and [104], and image captioning [105, 106] have all proven quite effective with the seq2seq models.”
MethodTip-Adapter: Training-free CLIP-Adapter for Better Vision-Language ModelingPreprint
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03930 · Cited by 472
“As vision and language usually contain complementary information, joint learning of vision and language representations has been proven to be quite effective on multimodality tasks, such as Visual Question Answering [1, 2, 18, 33], Image Captioning [29, 67], and Referring Expression [68].”
BackgroundA Machine Learning-Oriented Survey on Tiny Machine Learning
Capogrosso, L., Cunico, F., Cheng, D. S., Fummi, F., & Cristani, M. (2024) · IEEE Access · doi:10.1109/access.2024.3365349 · Cited by 209
“Beyond translation tasks, the attention mechanism has been successfully employed in natural language processing [113], image captioning [112], speech recognition [111], and more.”
BackgroundMethodVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Spatially-Adaptive Image Restoration using Distortion-Guided Networks
Purohit, K., Suin, M., Rajagopalan, A. N., & Boddeti, V. N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00231 · Cited by 156
“image captioning [18]) have shown that this behavior is not practically achieved.”
BackgroundGRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features
Nguyen, V.-Q., Suganuma, M., & Okatani, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_10 · Cited by 152
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders
Messina, N., Amato, G. M., Esuli, A., Falchi, F., Gennaro, C., & Marchand‐Maillet, S. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3451390 · Cited by 163
“Recently, many attempts to merge the two worlds have been explored, and state-of-the-art results have been obtained in many of these tasks, including visual question answering [2, 12, 42], image captioning [6, 14, 40, 49] and image-text matching [5, 10, 22, 33].”
BackgroundMore Grounded Image Captioning by Distilling Image-Text Matching Model
Zhou, Y., Wang, M., Liu, D., Hu, Z., & Zhang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00483 · Cited by 147
“Attention mechanisms have been operated on uniform spatial grids [56, 35], semantic metadata [61, 57, 12], and object-level regions [3, 18, 60, 64].”
BackgroundCoSMo: Content-Style Modulation for Image Retrieval with Text Feedback
Lee, S., Kim, D.-W., & Han, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00086 · Cited by 144
“In the compositor, we first calculate the underlying style information represented by ( µ x r , σ x r ) and remove it from the image feature x r by applying instance normalization, which is given by Then, the contents and style are modified based on the text via CM followed by SM.”
MethodContext-aware Attentional Pooling (CAP) for Fine-grained Visual Classification
Behera, A., Wharton, Z., Hewage, P., & Bera, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16176 · Cited by 10
“Thus, it can benefit to a wide variety of applications such as image captioning (Herdade et al. 2019; Huang et al. 2019a; Li et al. 2019), expert-level image recognition (Valan et al. 2019; Krause et al. 2016), and so on.”
BackgroundImage Captioning Through Image Transformer
He, S., Liao, W., Tavakoli, H. R., Yang, M. Y., Rosenhahn, B., & Pugeault, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_10 · Cited by 147
“In the ablation study, we use AoA [12] as a strong baseline 6 (with a single multihead dot-product attention module per layer), which add the gated linear layer [27] on top of the multi-head attention.”
BackgroundMethodInfluentialInjecting Semantic Concepts into End-to-End Image Captioning
Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01748 · Cited by 146
“These models [11, 27,53,81] all come unanimously with heavy computational burdens and extra learnable parameters.”
BackgroundTowards local visual modeling for image captioning
Ma, Y., Ji, J., Sun, X., Zhou, Y., & Ji, R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109420 · Cited by 139
“The compared models include: SCST [32], Up-Down [5], RFNet [34], GCN-LSTM [35], SGAE [36], AoANet [9], ETA [38], ORT [39], Transformer [19], M2Transformer [2], XTransformer [8], RSTNet [11] and DLCT [12].”
BackgroundMethodInfluentialComprehending and Ordering Semantics for Image Captioning
Li, Y., Pan, Y., Yao, T., & Mei, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01746 · Cited by 139
“Then, a series of innovations that employ visual attention over these fine-grained semantics [6,10] are present to strengthen vision-language interaction.”
BackgroundMethodDeconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 136
Oscar: Object-Semantics Aligned Pre-training for Vision-Language TasksPreprint
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.ldk.2021.36 · Cited by 129
Semantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 139
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128
Multistage attention network for image inpainting
Wang, N., Ma, S., Li, J., Zhang, Y., & Zhang, L. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107448 · Cited by 125
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 130
“8 AoANet [22] 42.”
Graph-Based Multimodal Sequential Embedding for Sign Language Translation
Tang, S., Guo, D., Hong, R., & Wang, M. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3117124 · Cited by 112
“In addition, we adopt the semantic metrics used in the fields of NLP [58], NMT [59] and image captioning [60], such as CIDEr, BLEU, ROUGE-L and METEOR.”
MethodDual Global Enhanced Transformer for image captioning
Xian, T., Li, Z., Zhang, C., & Ma, H. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.01.011 · Cited by 113
Changes to Captions: An Attentive Network for Remote Sensing Change Captioning
Chang, S., & Ghamisi, P. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3328224 · Cited by 127
“To improve the realism of generated descriptions, attention mechanisms have been incorporated into the encoder-decoder paradigm, which is widely used in image captioning tasks [28]–[30].”
MethodCross on Cross Attention: Deep Fusion Transformer for Image Captioning
Zhang, J., Xie, Y., Ding, W., & Wang, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3243725 · Cited by 109
“Up-Dwon [24], AoANet [20] and X-Transformer [27] use Faster R-CNN with ResNet101 from Up-Down (Up-Down-R101) to extract region features and facilitate the captioning model by improving the attention mechanism.”
BackgroundMethodBeyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
Kuo, C.-W., & Kira, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01744 · Cited by 99
“Because of its great success, encoding the input image with a set of detected objects has become a standard approach in recent VL works [1, 17, 19, 20, 25, 39, 52], as well as in VL pretraining [7, 28, 32, 45, 49].”
BackgroundThe Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis
Barraco, M., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00512 · Cited by 98
“As for the language model, in earlier works it was implemented as a recurrent neural network [15,18,20,36], while more recent approaches employ Transformer-based fully-attentive models [5,8,28,56].”
MethodHigh-Order Interaction Learning for Image Captioning
Wang, Y., Xu, N., Liu, A.-A., Li, W., & Zhang, Y. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3121062 · Cited by 95
“Following the idea of using objects to build the visual representation in most captioning works [16]–[18], [37], we remove the relationship features and only use the mean pooling of object features to constitute f : As shown in Table IV, Obj-Feat and our model outperform FC-Feat, which indicates…”
BackgroundMethodInfluentialFine-grained Iterative Attention Network for Temporal Language Localization in Videos
Qu, X., Tang, P., Zou, Z., Cheng, Y., Dong, J., Zhou, P., & Xu, Z. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414053 · Cited by 92
“Furthermore, to obtain more distinguishing features, we employ an information gate [14] on CMA to refine the output information as noisy information could be captured by partial heads.”
MethodTrends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 146
Global Visual Feature and Linguistic State Guided Attention for Remote Sensing Image Captioning
Zhang, Z., Zhang, W., Yan, M., Gao, X., Fu, K., & Sun, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3132095 · Cited by 96
Toward Automatic Audio Description Generation for Accessible Videos
Wang, Y., Liang, W., Huang, H., Zhang, Y., Li, D., & Yu, L.-F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445347 · Cited by 87
“Similar to image captioning, it requires not only recognizing salient objects, but also understanding actions and interactions [26].”
BackgroundHierarchical Feature Fusion With Mixed Convolution Attention for Single Image Dehazing
Zhang, X., Wang, J., Wang, T., & Jiang, R. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3067062 · Cited by 86
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 85
“We examine the captions generated by six image captioning models: (1) FC [59] is a simple sequence encoder that takes in image features encoded by a CNN; (2) Att2in [59] is similar but images are encoded using spatial features; (3) DiscCap [50] further adds a loss term to encourage discriminability; (4-6) Transformer…”
BackgroundMethodInfluentialTransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning
Chen, S., Hong, Z., Hou, W., Xie, G.-S., Song, Y., Zhao, J. C., You, X., Yan, S., & Shao, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229526 · Cited by 87
“Some methods [57], [59], [60] have also shown that the Transformer can better capture the relationship between various modals (e.g., visual features and language) in parallel during training.”
MethodAugmented reality, deep learning and vision-language query system for construction worker safety
Chen, H., Hou, L., Wu, S., Zhang, G., Zou, Y., Moon, S., & Bhuiyan, M. A. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105158 · Cited by 90
Incorporating attentive multi-scale context information for image captioning
Prudviraj, J., Yenduri, S., & Mohan, C. K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11895-9 · Cited by 85
“In addition, our approach learns to align object level & semantic level features while generating natural sentences and shows superior performance over the previous state of the approaches which exploits semantic alignment embedding [69] and self-attention mechanisms [25,14].”
BackgroundTransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
He, D., Zhao, Y., Luo, J., Hui, T., Huang, S., Zhang, A., & Liu, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475397 · Cited by 88
“Specific tasks like visual question answering [24, 32, 33], image captioning [2, 9, 14], image-text matching [20, 21, 29], visual relation detection [8] and scene graph generation [36] have been widely studied in the past few years.”
BackgroundMulti-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 105
X-Pruner: eXplainable Pruning for Vision Transformers
Lu, Y., & Xiang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02333 · Cited by 96
“For the second issue, previous studies resort to identifying unimportant weights by various importance metrics, including magnitude-based, gradient-based [12, 19], and mask-based [27].”
BackgroundRecurrent Attention and Semantic Gate for Remote Sensing Image Captioning
Li, Y., Zhang, X., Gu, J., Li, C., Wang, X., Tang, X., & Jiao, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3102590 · Cited by 87
BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detection
Yang, X., Qian, Y., Zhu, H., Wang, C., & Yang, M. Z. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811999 · Cited by 79
“The idea of attention mechanisms, which is inspired by the human visual attention mechanisms, is widely applied in various computer vision tasks, including image classification [21], [22], semantic segmentation [23], [24], image captioning [25], [26], etc.”
BackgroundVisual News: Benchmark and Challenges in News Image Captioning
Liu, F., Wang, Y., Wang, T., & Ordóñez, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.542 · Cited by 133
“In addition, we have the "Attention on Attention" (AoA) module (Huang et al., 2019) to assist the generation of attended information: where (cid:12) represents the element-wise multiplication operation and σ is the sigmoid function.”
S2 Transformer for Image Captioning
Zeng, P., Zhang, H., Song, J., & Gao, L. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/224 · Cited by 78
M-FFN: multi-scale feature fusion network for image captioning
Prudviraj, J., Vishnu, C., & Mohan, C. K. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03463-x · Cited by 78
“Further, the dilated network, dilated network + spatial-semantic attention (SSA), and dilated network + STN report the performance of 120.9, 129.7, and 136.2, respectively, and showing better performance than the recent state-of-the-art approaches like Recall [54], AOA [55], and M 2 T [56].”
BackgroundInfluentialDeep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 79
Adaptive Path Selection for Dynamic Image Captioning
Xian, T., Li, Z., Tang, Z., & Ma, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3155795 · Cited by 76
“We optimize the following cross-entropy (XE) loss : Following previous works [6], [9], [53], we use the CIDEr-D score as reward to finetune the model.”
BackgroundMethodInfluentialDIFNet: Boosting Visual Information Flow for Image Captioning
Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., & Ji, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01749 · Cited by 77
“The encoder-decoder framework has been widely adopted by image captioning models [2, 5, 9].”
MethodInfluentialAn Improved Attention for Visual Question Answering
Rahman, T., Chou, S.-H., Sigal, L., & Carenini, G. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00181 · Cited by 74
“Motivated by Huang et al. [10], in this paper we leverage the idea of Attention on Attention (AoA) module to address the above mentioned limitation.”
InfluentialGeometry Attention Transformer with position-aware LSTMs for image captioning
Wang, C., Shen, Y., & Ji, L. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117174 · Cited by 74
“Similarly, our GAT also outperforms the AOANet [20], in terms of almost all metrics except the CIDEr of Cross-Entropy.”
BackgroundMethodInfluentialVisuals to Text: A Comprehensive Review on Automatic Image Captioning
Ming, Y., Hu, N., Fan, C., Feng, F., Zhou, J., & Yu, H. L. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105734 · Cited by 85
Show, Edit and Tell: A Framework for Editing Image Captions
Sammani, F., & Melas-Kyriazi, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00486 · Cited by 73
“[23], where we employ copy mechanisms to better take advantage of the information in the existing caption. Compared to both [23] and the state-ofthe-art in image caption generation (discussed above) [8], we show significant performance improvements across image captioning metrics. 3. Proposed Methodology Our model cons…”
BackgroundMethodInfluentialTowards Accurate Text-based Image Captioning with Content Diversity Exploration
Xu, G., Niu, S., Tan, M., Luo, Y., Du, Q., & Wu, Q. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01245 · Cited by 74
“From the table, the BUTD and AoANet, standard image captioning models, show poor performances on the validation set since they fail to describe the texts in images.”
BackgroundMethodInfluentialShow, Deconfound and Tell: Image Captioning with Causal Inference
Liu, B., Wang, D., Yang, X., Zhou, Y. Z., Yao, R., Shao, Z., & Zhao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01751 · Cited by 75
“In Table 2, we compare our CIIC with the SOTA models on the offline COCO Karpathy test split, including SCST [34], Up-Down [2], RFNet [19], GCN-LSTM [46], SGAE [42], ORT [15], AoANet [16], M 2 Transformer [8], Transformer+CATT [45] and X-Transformer [28].”
MethodTransferable Decoding with Visual Entities for Zero-Shot Image Captioning
Fei, J., Wang, T., Zhang, J., He, Z., Wang, C., & Zheng, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00291 · Cited by 81
“Various attention mechanisms [50, 21, 51, 7, 31] are commonly designed to capture vision-language alignment cues.”
BackgroundMethodXGPT: Cross-modal Generative Pre-Training for Image Captioning
Xia, Q., Huang, H., Duan, N., Zhang, D., Ji, L., Sui, Z., Cui, E., Bharti, T., & Zhou, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-88480-2_63 · Cited by 70
“5 - - - AoANet [24] - - - - 119.”
Generating Image Captions Using Bahdanau Attention Mechanism and Transfer Learning
Ayoub, S., Gulzar, Y., Reegu, F. A., & Turaev, S. (2022) · Symmetry · doi:10.3390/sym14122681 · Cited by 69
“Huang et al. [31] proposed a novel framework called Attention on Attention.”
Length-Controllable Image Captioning
Deng, C., Ding, N., Tan, M., & Wu, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_42 · Cited by 70
“Specifically, we implement it on AoANet [20] and VLP [53], which are the most recent SOTA image captioning models of the two decoder categories, respectively.”
BackgroundMethodInfluentialBoosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph
Zhao, W., & Wu, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301279 · Cited by 73
“Boosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph Wentian Zhao and Xinxiao Wu , Member, IEEE Abstract —Entity-aware image captioning aims to describe named entities and events related to the image by utilizing the background knowledge in the associated article.”
BackgroundText to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 78
“[34] proposed an attention on attention (AoA) module for image captioning.”
BackgroundActive Sensing for Communications by Learning
Sohrabi, F., Jiang, T., Cui, W., & Yu, W. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3155496 · Cited by 75
“It has been widely used in areas including sequence-to-sequence machine translation [26]–[29], image captioning [30], [31], and reinforcement learning [32].”
MethodVIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Hu, X., Yin, X. P., Lin, K., Zhang, L., Gao, J., Wang, L., & Liu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16249 · Cited by 77
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 67
“have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
BackgroundDouble-Stream Position Learning Transformer Network for Image Captioning
Jiang, W., Zhou, W., & Hu, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3181490 · Cited by 66
Underwater image captioning: Challenges, models, and datasets
Li, H., Wang, H., Zhang, Y., Li, L., & Ren, P. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.12.002 · Cited by 66
Memory-Based Augmentation Network for Video Captioning
Jing, S., Zhang, H., Zeng, P., Gao, L., Song, J., & Shen, H. T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3295098 · Cited by 65
Assessing Image Quality Issues for Real-World Problems
Chiu, T.-Y., Zhao, Y., & Gurari, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00370 · Cited by 63
“AoANet [19] full training set 63.”
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
