Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DropToken reduces training complexity via random dropping a portion of video and audio tokens from input sequence during training.
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Control of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · cited 1× · see citing works
“Several works extend the idea of employing self-attention to some tasks [40, 19] in computer vision, which inspires us to apply self-attention to image captioning to model relationships among objects in an image.”
Background(2004) · Cited by 9,718 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
(2005) · Cited by 4,152 · see citing works
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,682 · see citing works
Language Modeling with Gated Convolutional NetworksPreprint
Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1612.08083 · Cited by 3,097 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.667 · Cited by 2,073 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · see citing works
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Relation Networks for Object Detection
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00378 · Cited by 1,524 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
The Dynamic Representation of Scenes
Rensink, R. A. (2000) · Visual Cognition · doi:10.1080/135062800394667 · Cited by 1,079 · see citing works
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_42 · Cited by 962 · see citing works
Gao, Y., Beijbom, O., Zhang, N., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.41 · Cited by 912 · see citing works
Auto-Encoding Scene Graphs for Image Captioning
Yang, X., Tang, K., Zhang, H., & Cai, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01094 · Cited by 852 · see citing works
BabyTalk: Understanding and Generating Simple Image Descriptions
Kulkarni, G. A., Premraj, V., Ordóñez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.162 · Cited by 839 · see citing works
(2017) · International Conference on Machine Learning · Cited by 758 · see citing works
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 752 · see citing works
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadène, R., Cord, M., & Thome, N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 733 · see citing works
Hadamard Product for Low-rank Bilinear PoolingPreprint
Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04325 · Cited by 604 · see citing works
Semantic Compositional Networks for Visual Captioning
Gan, Z., Gan, C., He, X., Pu, Y., Tran, K., Gao, J., Carin, L., & Deng, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.127 · Cited by 462 · see citing works
Midge: Generating Image Descriptions From Computer Vision Detections
Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T. L., & Daumé, H. (2012) · Cited by 421 · see citing works
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00680 · Cited by 415 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual GroundingPreprint
Fukui, A., Park, D. H., Yang, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01847 · Cited by 388 · see citing works
Corpus-Guided Sentence Generation of Natural Images
Yang, Y., Teo, C. L., Daumé, H., & Aloimonos, Y. (2011) · Cited by 367 · see citing works
I2T: Image Parsing to Text Description
Yao, B., Yang, X., Lin, L., Lee, M. W., & Zhu, S. C. (2010) · Proceedings of the IEEE · doi:10.1109/jproc.2010.2050411 · Cited by 310 · see citing works
Recurrent Fusion Network for Image Captioning
Jiang, W., Ma, L., Jiang, Y., Liu, W., & Zhang, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_31 · Cited by 303 · see citing works
StyleNet: Generating Attractive Visual Captions with Styles
Gan, C., Gan, Z., He, X., Gao, J., & Deng, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.108 · Cited by 287 · see citing works
C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection
Wan, F., Liu, C., Ke, W., Ji, X., Jiao, J., & Ye, Q. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00230 · Cited by 257 · see citing works
Multi-level Attention Networks for Visual Question Answering
Yu, D., Fu, J., Mei, T., & Rui, Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.446 · Cited by 233 · see citing works
Review Networks for Caption GenerationPreprint
Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07912 · Cited by 230 · see citing works
Less Is More: Picking Informative Frames for Video Captioning
Chen, Y., Wang, S., Zhang, W., & Huang, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_22 · Cited by 222 · see citing works
Socher, R., & Fei-Fei, L. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540112 · Cited by 221 · see citing works
Min-Entropy Latent Model for Weakly Supervised Object Detection
Wan, F., Wei, P., Han, Z., Jiao, J., & Ye, Q. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2898858 · Cited by 87 · see citing works
C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object DetectionPreprint
Wan, F., Liu, C., Ke, W., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05647 · Cited by 30 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works
“2 AoANet [11] 37.”
ResultMeshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259 · see citing works
“Noticeably, when using four models our approach achieves the best performance according to all metrics, with an increase of 2.5 CIDEr points with respect to the current state of the art [14].”
BackgroundMethodInfluentialMultimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
“graph [228], video to caption [229], [230], [231], [232], image to caption [233], [234], [235], [236], [237], text to speech [238], text to image [205], [239], text to shape [240], RGB to 3D human”
BackgroundVinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 965 · see citing works
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
X-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720 · see citing works
“Later on, the most recently proposed attention on attention module [12] enhances visual attention by further measuring the relevance between the attention result and the query.”
BackgroundThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
“To help the agent understand the information contained within images, a straightforward approach is to generate corresponding text descriptions for image inputs, known as image captioning [203–207].”
Self-citationConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
“State-of-the-art V+L models heavily rely on self-attention mechanisms [73] or similar com-5 ponents that allow free information flow among visual regions, and image captioning is no exception [66, 80, 19, 32, 23].”
BackgroundTip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification
Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_29 · Cited by 464 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · see citing works
AiATrack: Attention in Attention for Transformer Visual Tracking
Gao, S., Zhou, C., Ma, C., Wang, X., & Yuan, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_9 · Cited by 396 · see citing works
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring
Suin, M., Purohit, K., & Rajagopalan, A. N. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00366 · Cited by 323 · see citing works
“(6) can be intuitively compared to [4], where similar gated-enhancement technique is used to refine the result by elementwise-multiplication with an attention mask that helps in propagating only the relevant information.”
MethodDual-level Collaborative Transformer for Image Captioning
Luo, Y., Ji, J., Sun, X., Cao, L., Wu, Y., Huang, F., Lin, C.-W., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16328 · Cited by 343 · see citing works
“To explain, most recent methods in image captioning (Huang et al. 2019; Cornia et al. 2020; Pan et al. 2020) use the self-attention modules to model the relationships of visual features.”
BackgroundMethodInfluentialFrom Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373 · see citing works
“Some works adopted the self-attention operator in place of the additive attention one in LSTM-based language models [79], [80], [83], [107].”
BackgroundRSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words
Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01521 · Cited by 297 · see citing works
“[16], GCN-LSTM [43], SGAE [42], ORT [12], AoANet [13], M(2) Transformer [6], and X-Transformer [28].”
BackgroundInfluentialWang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291 · see citing works
“AoANet [25] proposed an “Attention on Attention” module on feature encoder and caption decoder for refining with the self-attention mechanism.”
MethodFine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention
Huynh, D., & Elhamifar, E. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00454 · Cited by 262 · see citing works
“Attention can also be recursively learned in a hierarchical fashion [39, 40].”
BackgroundNormalized and Geometry-Aware Self-Attention Network for Image Captioning
Guo, L., Liu, J., Zhu, X., Yao, P., Lu, S., & Lu, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01034 · Cited by 259 · see citing works
“We compare NGSAN with the state-of-the-art methods, including Up-Down [2], CAVP [23], SGAE [42], VSUA [14], ORT [15], AoANet [17], and MT [43].”
MethodUnifying Vision-and-Language Tasks via Text GenerationPreprint
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 268 · see citing works
“XGPT uses AoA module (Huang et al., 2019) as visual encoder.”
MethodTextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_44 · Cited by 242 · see citing works
“9 3 AoANet [18] COCO 18.”
BackgroundMethodInfluentialExploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
Yuan, Z., Zhang, W., Fu, K., Li, X., Deng, C., Wang, H., & Sun, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3078451 · Cited by 245 · see citing works
“2) Fusion Attention: The motivation comes from attention on attention (AoA) [43].”
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 272 · see citing works
Radiology report generation with a learned knowledge base and multi-modal alignment
Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S. K., & Xiao, L. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102798 · Cited by 199 · see citing works
Deep Learning Approaches on Image Captioning: A Review
Ghandi, T., Pourreza, H. R., & Mahyar, H. (2023) · ACM Computing Surveys · doi:10.1145/3617592 · Cited by 205 · see citing works
“Huang et al. [ 59 ] have introduced a new attention-based structure containing one more level of attention.”
BackgroundMethodInfluentialVisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01750 · Cited by 205 · see citing works
“• AoA Transformer, which inserts an attention-on-attention (AoA) module [24] into every transformer layer, as depicted by Figure 3 (b).”
BackgroundMethodInfluentialSelf-citationJi, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16258 · Cited by 208 · see citing works
“…(Vaswani et al. 2017), where the self-attention is efficiently utilized to capture the correlations among the regions and words (Liu et al. 2019; Huang et al. 2019; Li et al. 2019a; Herdade et al. 2019; Cornia et al. 2020). is region-biased, which only focuses on the region-level information…”
BackgroundMethodInfluentialImage Search With Text Feedback by Visiolinguistic Attention Learning
Chen, Y., Gong, S., & Bazzani, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00307 · Cited by 191 · see citing works
“Recently, several concurrent works [59, 56, 35] extend the pre-training strategies from BERT [9] to learn the latent compositional representations, which jointly represent images and descriptive texts for solving VQA, captioning, or image-text matching.”
BackgroundMethodInfluentialCaptioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182 · see citing works
“For instance, we observe better performance when UpDown [8] and AoANet [30] are trained from scratch on VizWiz-Captions rather than fine-tuned from models pretrained on MSCOCO-Captions, and vice versa for SGAE [56].”
MethodInfluentialCausal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175 · see citing works
TransZero: Attribute-Guided Transformer for Zero-Shot Learning
Chen, S., Hong, Z., Liu, Y., Xie, G.-S., Sun, B., Li, H., Peng, Q., Lü, K., & You, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19909 · Cited by 174 · see citing works
“Some methods (Gabeur et al. 2020; Cornia et al. 2020; Huang et al. 2019; Pan et al. 2020) have also shown that the transformer architecture can better capture the relationship between visual features and process sequences in parallel during training.”
MethodAsymmetric Feature Fusion Network for Hyperspectral and SAR Image Classification
Li, W., Gao, Y., Zhang, M., Tao, R., & Du, Q. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3149394 · Cited by 170 · see citing works
End-to-End Transformer Based Model for Image Captioning
Wang, Y., Xu, J., & Sun, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20160 · Cited by 169 · see citing works
Transformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 177 · see citing works
“Machine translation [101, 102], text summarization [103] and [104], and image captioning [105, 106] have all proven quite effective with the seq2seq models.”
MethodTip-Adapter: Training-free CLIP-Adapter for Better Vision-Language ModelingPreprint
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03930 · Cited by 285 · see citing works
“As vision and language usually contain complementary information, joint learning of vision and language representations has been proven to be quite effective on multimodality tasks, such as Visual Question Answering [1, 2, 18, 33], Image Captioning [29, 67], and Referring Expression [68].”
BackgroundA Machine Learning-Oriented Survey on Tiny Machine Learning
Capogrosso, L., Cunico, F., Cheng, D. S., Fummi, F., & Cristani, M. (2024) · IEEE Access · doi:10.1109/access.2024.3365349 · Cited by 182 · see citing works
“Beyond translation tasks, the attention mechanism has been successfully employed in natural language processing [113], image captioning [112], speech recognition [111], and more.”
BackgroundMethodVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Spatially-Adaptive Image Restoration using Distortion-Guided Networks
Purohit, K., Suin, M., Rajagopalan, A. N., & Boddeti, V. N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00231 · Cited by 152 · see citing works
“image captioning [18]) have shown that this behavior is not practically achieved.”
BackgroundGRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features
Nguyen, V.-Q., Suganuma, M., & Okatani, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_10 · Cited by 152 · see citing works
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders
Messina, N., Amato, G. M., Esuli, A., Falchi, F., Gennaro, C., & Marchand‐Maillet, S. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3451390 · Cited by 158 · see citing works
“Recently, many attempts to merge the two worlds have been explored, and state-of-the-art results have been obtained in many of these tasks, including visual question answering [2, 12, 42], image captioning [6, 14, 40, 49] and image-text matching [5, 10, 22, 33].”
BackgroundMore Grounded Image Captioning by Distilling Image-Text Matching Model
Zhou, Y., Wang, M., Liu, D., Hu, Z., & Zhang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00483 · Cited by 144 · see citing works
“Attention mechanisms have been operated on uniform spatial grids [56, 35], semantic metadata [61, 57, 12], and object-level regions [3, 18, 60, 64].”
BackgroundCoSMo: Content-Style Modulation for Image Retrieval with Text Feedback
Lee, S., Kim, D.-W., & Han, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00086 · Cited by 144 · see citing works
“In the compositor, we first calculate the underlying style information represented by ( µ x r , σ x r ) and remove it from the image feature x r by applying instance normalization, which is given by Then, the contents and style are modified based on the text via CM followed by SM.”
MethodContext-aware Attentional Pooling (CAP) for Fine-grained Visual Classification
Behera, A., Wharton, Z., Hewage, P., & Bera, A. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i2.16176 · Cited by 141 · see citing works
“Thus, it can benefit to a wide variety of applications such as image captioning (Herdade et al. 2019; Huang et al. 2019a; Li et al. 2019), expert-level image recognition (Valan et al. 2019; Krause et al. 2016), and so on.”
BackgroundImage Captioning Through Image Transformer
He, S., Liao, W., Tavakoli, H. R., Yang, M. Y., Rosenhahn, B., & Pugeault, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_10 · Cited by 140 · see citing works
“In the ablation study, we use AoA [12] as a strong baseline 6 (with a single multihead dot-product attention module per layer), which add the gated linear layer [27] on top of the multi-head attention.”
BackgroundMethodInfluentialInjecting Semantic Concepts into End-to-End Image Captioning
Fang, Z., Wang, J., Hu, X., Liang, L., Gan, Z., Wang, L., Yang, Y., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01748 · Cited by 141 · see citing works
“These models [11, 27,53,81] all come unanimously with heavy computational burdens and extra learnable parameters.”
BackgroundTowards local visual modeling for image captioning
Ma, Y., Ji, J., Sun, X., Zhou, Y., & Ji, R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109420 · Cited by 139 · see citing works
“The compared models include: SCST [32], Up-Down [5], RFNet [34], GCN-LSTM [35], SGAE [36], AoANet [9], ETA [38], ORT [39], Transformer [19], M2Transformer [2], XTransformer [8], RSTNet [11] and DLCT [12].”
BackgroundMethodInfluentialComprehending and Ordering Semantics for Image Captioning
Li, Y., Pan, Y., Yao, T., & Mei, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01746 · Cited by 138 · see citing works
“Then, a series of innovations that employ visual attention over these fine-grained semantics [6,10] are present to strengthen vision-language interaction.”
BackgroundMethodDeconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 130 · see citing works
Oscar: Object-Semantics Aligned Pre-training for Vision-Language TasksPreprint
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.ldk.2021.36 · Cited by 129 · see citing works
Semantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 128 · see citing works
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128 · see citing works
Multistage attention network for image inpainting
Wang, N., Ma, S., Li, J., Zhang, Y., & Zhang, L. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107448 · Cited by 123 · see citing works
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 130 · see citing works
“8 AoANet [22] 42.”
Graph-Based Multimodal Sequential Embedding for Sign Language Translation
Tang, S., Guo, D., Hong, R., & Wang, M. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3117124 · Cited by 110 · see citing works
“In addition, we adopt the semantic metrics used in the fields of NLP [58], NMT [59] and image captioning [60], such as CIDEr, BLEU, ROUGE-L and METEOR.”
MethodDual Global Enhanced Transformer for image captioning
Xian, T., Li, Z., Zhang, C., & Ma, H. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.01.011 · Cited by 109 · see citing works
Changes to Captions: An Attentive Network for Remote Sensing Change Captioning
Chang, S., & Ghamisi, P. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3328224 · Cited by 112 · see citing works
“To improve the realism of generated descriptions, attention mechanisms have been incorporated into the encoder-decoder paradigm, which is widely used in image captioning tasks [28]–[30].”
MethodCross on Cross Attention: Deep Fusion Transformer for Image Captioning
Zhang, J., Xie, Y., Ding, W., & Wang, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3243725 · Cited by 107 · see citing works
“Up-Dwon [24], AoANet [20] and X-Transformer [27] use Faster R-CNN with ResNet101 from Up-Down (Up-Down-R101) to extract region features and facilitate the captioning model by improving the attention mechanism.”
BackgroundMethodBeyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning
Kuo, C.-W., & Kira, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01744 · Cited by 99 · see citing works
“Because of its great success, encoding the input image with a set of detected objects has become a standard approach in recent VL works [1, 17, 19, 20, 25, 39, 52], as well as in VL pretraining [7, 28, 32, 45, 49].”
BackgroundThe Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis
Barraco, M., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00512 · Cited by 97 · see citing works
“As for the language model, in earlier works it was implemented as a recurrent neural network [15,18,20,36], while more recent approaches employ Transformer-based fully-attentive models [5,8,28,56].”
MethodHigh-Order Interaction Learning for Image Captioning
Wang, Y., Xu, N., Liu, A.-A., Li, W., & Zhang, Y. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3121062 · Cited by 95 · see citing works
“Following the idea of using objects to build the visual representation in most captioning works [16]–[18], [37], we remove the relationship features and only use the mean pooling of object features to constitute f : As shown in Table IV, Obj-Feat and our model outperform FC-Feat, which indicates…”
BackgroundMethodInfluentialFine-grained Iterative Attention Network for Temporal Language Localization in Videos
Qu, X., Tang, P., Zou, Z., Cheng, Y., Dong, J., Zhou, P., & Xu, Z. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414053 · Cited by 92 · see citing works
“Furthermore, to obtain more distinguishing features, we employ an information gate [14] on CMA to refine the output information as noisy information could be captured by partial heads.”
MethodTrends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 128 · see citing works
Global Visual Feature and Linguistic State Guided Attention for Remote Sensing Image Captioning
Zhang, Z., Zhang, W., Yan, M., Gao, X., Fu, K., & Sun, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3132095 · Cited by 92 · see citing works
Toward Automatic Audio Description Generation for Accessible Videos
Wang, Y., Liang, W., Huang, H., Zhang, Y., Li, D., & Yu, L.-F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445347 · Cited by 87 · see citing works
“Similar to image captioning, it requires not only recognizing salient objects, but also understanding actions and interactions [26].”
BackgroundHierarchical Feature Fusion With Mixed Convolution Attention for Single Image Dehazing
Zhang, X., Wang, J., Wang, T., & Jiang, R. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3067062 · Cited by 86 · see citing works
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 84 · see citing works
“We examine the captions generated by six image captioning models: (1) FC [59] is a simple sequence encoder that takes in image features encoded by a CNN; (2) Att2in [59] is similar but images are encoded using spatial features; (3) DiscCap [50] further adds a loss term to encourage discriminability; (4-6) Transformer…”
BackgroundMethodInfluentialTransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning
Chen, S., Hong, Z., Hou, W., Xie, G.-S., Song, Y., Zhao, J. C., You, X., Yan, S., & Shao, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229526 · Cited by 83 · see citing works
“Some methods [57], [59], [60] have also shown that the Transformer can better capture the relationship between various modals (e.g., visual features and language) in parallel during training.”
MethodAugmented reality, deep learning and vision-language query system for construction worker safety
Chen, H., Hou, L., Wu, S., Zhang, G., Zou, Y., Moon, S., & Bhuiyan, M. A. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105158 · Cited by 90 · see citing works
Incorporating attentive multi-scale context information for image captioning
Prudviraj, J., Yenduri, S., & Mohan, C. K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11895-9 · Cited by 82 · see citing works
“In addition, our approach learns to align object level & semantic level features while generating natural sentences and shows superior performance over the previous state of the approaches which exploits semantic alignment embedding [69] and self-attention mechanisms [25,14].”
BackgroundTransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
He, D., Zhao, Y., Luo, J., Hui, T., Huang, S., Zhang, A., & Liu, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475397 · Cited by 81 · see citing works
“Specific tasks like visual question answering [24, 32, 33], image captioning [2, 9, 14], image-text matching [20, 21, 29], visual relation detection [8] and scene graph generation [36] have been widely studied in the past few years.”
BackgroundMulti-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 83 · see citing works
X-Pruner: eXplainable Pruning for Vision Transformers
Lu, Y., & Xiang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02333 · Cited by 80 · see citing works
“For the second issue, previous studies resort to identifying unimportant weights by various importance metrics, including magnitude-based, gradient-based [12, 19], and mask-based [27].”
BackgroundRecurrent Attention and Semantic Gate for Remote Sensing Image Captioning
Li, Y., Zhang, X., Gu, J., Li, C., Wang, X., Tang, X., & Jiao, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3102590 · Cited by 80 · see citing works
BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detection
Yang, X., Qian, Y., Zhu, H., Wang, C., & Yang, M. Z. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811999 · Cited by 79 · see citing works
“The idea of attention mechanisms, which is inspired by the human visual attention mechanisms, is widely applied in various computer vision tasks, including image classification [21], [22], semantic segmentation [23], [24], image captioning [25], [26], etc.”
BackgroundVisual News: Benchmark and Challenges in News Image Captioning
Liu, F., Wang, Y., Wang, T., & Ordóñez, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.542 · Cited by 78 · see citing works
“In addition, we have the "Attention on Attention" (AoA) module (Huang et al., 2019) to assist the generation of attended information: where (cid:12) represents the element-wise multiplication operation and σ is the sigmoid function.”
S2 Transformer for Image Captioning
Zeng, P., Zhang, H., Song, J., & Gao, L. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/224 · Cited by 78 · see citing works
M-FFN: multi-scale feature fusion network for image captioning
Prudviraj, J., Vishnu, C., & Mohan, C. K. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03463-x · Cited by 78 · see citing works
“Further, the dilated network, dilated network + spatial-semantic attention (SSA), and dilated network + STN report the performance of 120.9, 129.7, and 136.2, respectively, and showing better performance than the recent state-of-the-art approaches like Recall [54], AOA [55], and M 2 T [56].”
BackgroundInfluentialDeep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 78 · see citing works
Adaptive Path Selection for Dynamic Image Captioning
Xian, T., Li, Z., Tang, Z., & Ma, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3155795 · Cited by 75 · see citing works
“We optimize the following cross-entropy (XE) loss : Following previous works [6], [9], [53], we use the CIDEr-D score as reward to finetune the model.”
BackgroundMethodInfluentialDIFNet: Boosting Visual Information Flow for Image Captioning
Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., & Ji, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01749 · Cited by 74 · see citing works
“The encoder-decoder framework has been widely adopted by image captioning models [2, 5, 9].”
MethodInfluentialAn Improved Attention for Visual Question Answering
Rahman, T., Chou, S.-H., Sigal, L., & Carenini, G. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00181 · Cited by 74 · see citing works
“Motivated by Huang et al. [10], in this paper we leverage the idea of Attention on Attention (AoA) module to address the above mentioned limitation.”
InfluentialGeometry Attention Transformer with position-aware LSTMs for image captioning
Wang, C., Shen, Y., & Ji, L. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117174 · Cited by 74 · see citing works
“Similarly, our GAT also outperforms the AOANet [20], in terms of almost all metrics except the CIDEr of Cross-Entropy.”
BackgroundMethodInfluentialVisuals to Text: A Comprehensive Review on Automatic Image Captioning
Ming, Y., Hu, N., Fan, C., Feng, F., Zhou, J., & Yu, H. L. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105734 · Cited by 80 · see citing works
Show, Edit and Tell: A Framework for Editing Image Captions
Sammani, F., & Melas-Kyriazi, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00486 · Cited by 73 · see citing works
“[23], where we employ copy mechanisms to better take advantage of the information in the existing caption. Compared to both [23] and the state-ofthe-art in image caption generation (discussed above) [8], we show significant performance improvements across image captioning metrics. 3. Proposed Methodology Our model cons…”
BackgroundMethodInfluentialTowards Accurate Text-based Image Captioning with Content Diversity Exploration
Xu, G., Niu, S., Tan, M., Luo, Y., Du, Q., & Wu, Q. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01245 · Cited by 71 · see citing works
“From the table, the BUTD and AoANet, standard image captioning models, show poor performances on the validation set since they fail to describe the texts in images.”
BackgroundMethodInfluentialShow, Deconfound and Tell: Image Captioning with Causal Inference
Liu, B., Wang, D., Yang, X., Zhou, Y. Z., Yao, R., Shao, Z., & Zhao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01751 · Cited by 71 · see citing works
“In Table 2, we compare our CIIC with the SOTA models on the offline COCO Karpathy test split, including SCST [34], Up-Down [2], RFNet [19], GCN-LSTM [46], SGAE [42], ORT [15], AoANet [16], M 2 Transformer [8], Transformer+CATT [45] and X-Transformer [28].”
MethodTransferable Decoding with Visual Entities for Zero-Shot Image Captioning
Fei, J., Wang, T., Zhang, J., He, Z., Wang, C., & Zheng, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00291 · Cited by 72 · see citing works
“Various attention mechanisms [50, 21, 51, 7, 31] are commonly designed to capture vision-language alignment cues.”
BackgroundMethodXGPT: Cross-modal Generative Pre-Training for Image Captioning
Xia, Q., Huang, H., Duan, N., Zhang, D., Ji, L., Sui, Z., Cui, E., Bharti, T., & Zhou, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-88480-2_63 · Cited by 70 · see citing works
“5 - - - AoANet [24] - - - - 119.”
Generating Image Captions Using Bahdanau Attention Mechanism and Transfer Learning
Ayoub, S., Gulzar, Y., Reegu, F. A., & Turaev, S. (2022) · Symmetry · doi:10.3390/sym14122681 · Cited by 69 · see citing works
“Huang et al. [31] proposed a novel framework called Attention on Attention.”
Length-Controllable Image Captioning
Deng, C., Ding, N., Tan, M., & Wu, Q. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_42 · Cited by 69 · see citing works
“Specifically, we implement it on AoANet [20] and VLP [53], which are the most recent SOTA image captioning models of the two decoder categories, respectively.”
BackgroundMethodInfluentialBoosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph
Zhao, W., & Wu, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301279 · Cited by 73 · see citing works
“Boosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph Wentian Zhao and Xinxiao Wu , Member, IEEE Abstract —Entity-aware image captioning aims to describe named entities and events related to the image by utilizing the background knowledge in the associated article.”
BackgroundText to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 67 · see citing works
“[34] proposed an attention on attention (AoA) module for image captioning.”
BackgroundActive Sensing for Communications by Learning
Sohrabi, F., Jiang, T., Cui, W., & Yu, W. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3155496 · Cited by 68 · see citing works
“It has been widely used in areas including sequence-to-sequence machine translation [26]–[29], image captioning [30], [31], and reinforcement learning [32].”
MethodVIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Hu, X., Yin, X. P., Lin, K., Zhang, L., Gao, J., Wang, L., & Liu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16249 · Cited by 73 · see citing works
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 66 · see citing works
“have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
BackgroundDouble-Stream Position Learning Transformer Network for Image Captioning
Jiang, W., Zhou, W., & Hu, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3181490 · Cited by 65 · see citing works
Underwater image captioning: Challenges, models, and datasets
Li, H., Wang, H., Zhang, Y., Li, L., & Ren, P. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.12.002 · Cited by 64 · see citing works
Memory-Based Augmentation Network for Video Captioning
Jing, S., Zhang, H., Zeng, P., Gao, L., Song, J., & Shen, H. T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3295098 · Cited by 63 · see citing works
Assessing Image Quality Issues for Real-World Problems
Chiu, T.-Y., Zhao, Y., & Gurari, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00370 · Cited by 63 · see citing works
“AoANet [19] full training set 63.”
Dual Attention on Pyramid Feature Maps for Image Captioning
Yu, L., Zhang, J., & Wu, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3072479 · Cited by 62 · see citing works
“Different from the attention model [Huang et al., 2019a] mainly focuses on language generation, we try to explore visual feature representations for image captioning.”
BackgroundMethodInfluential
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2 · 2012: 1
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
