The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Cited by
Sorted by citations
Items 701–800 of 1,002
ragBERT: Relationship-aligned and grammar-wise BERT model for image captioning
Wang, H., Song, K., Jiang, X., & He, Z. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105105 · Cited by 4
Learning to Describe Scenes via Privacy-Aware Designed Optical Lens
Arguello, P., Lopez, J., Sánchez, K., Hinojosa, C. A., Morales, F. A. R., & Argüello, H. (2024) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2024.3426975 · Cited by 3
“Several approaches have been proposed to tackle this task, with some models relying on attention mechanisms to focus on salient regionsintheimage[5],[7],[21].”
BackgroundTransformer with multi-level grid features and depth pooling for image captioning
Bui, D. C., Nguyen, T., & Nguyen, K. T. T. M. (2024) · Machine Vision and Applications · doi:10.1007/s00138-024-01599-z · Cited by 3
“Most recent studies tackled the problem by proposing variants of Transformer-based models [7, 9–11 , 17– 23].”
InfluentialTransformer with token attention and attribute prediction for image captioning
Song, L., Wang, Y., Shi, L., Yu, J., Li, F., & Xiang, S. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.11.022 · Cited by 3
VTIENet: visual-text information enhancement network for image captioning
Yang, J., Wei, Y., Wang, R., & Xue, L. (2025) · Multimedia Systems · doi:10.1007/s00530-024-01658-5 · Cited by 3
“Many subsequent studies use object-level regional features as image visual feature input to generate more accurate image captions [2, 10–14].”
ScHiCAtt: Enhancing single-cell Hi-C data resolution using attention-based models
Menon, R., Chowdhury, H. M. A. M., & Oluwadare, O. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.02.031 · Cited by 3
@BENCH: Benchmarking Vision-Language Models for Human-centered Assistive Technology
Jiang, X., Zheng, J., Liu, R.-P., Li, J., Zhang, J., Matthiesen, S., & Stiefelhagen, R. (2024) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00387 · Cited by 7
“…works for each task are: MaskFormer [7], Mask2Former [6] and kMaX-DeepLab [44] for panoptic segmentation; BTS [21], DPT [32] and GLP [19] for depth estimation; ASTER [35], SEED [30] and MaskOCR [25] for OCR; VizWiz Cap, AoANet [13] for captioning, VizWiz VQA, S-VQA [18] and CS-VQA [15] for VQA.”
DMFormer: Dense Memory Linformer for Image Captioning
He, Y., & Jiang, Z. (2025) · Electronics · doi:10.3390/electronics14091716 · Cited by 3
“Several mainstream methods involved in the comparison include SCST [31], RFNet [40], Up-Down [41], GCN-LSTM [42], SGAE [37], ORT [38], AoANet [39], M2-Transformer [29], X-Transformer [24], RSTNet [43], DGET [44], GAT [10], ViTCAP [17], CATNet [45], MAENet [46], D2 Transformer [20], VaT [47],…”
Semantically Enhanced Dual Visual Fusion Transformer for accurate image captioning
Yang, J., Zhang, H., Wang, R., Xue, L., & Zhang, J. (2025) · The Visual Computer · doi:10.1007/s00371-025-03905-w · Cited by 3
“The compared models include: SCST [52], Up-Down [4], GCN-LSTM [61], ETA [39], ORT [40], AoANet [62], M 2 Transformer[63],GET[64],X-Transformer[65],TCIC[66], RSTNet [16], CATNet [67], DLCT [15], LG-MLFormer [68], RCT [69], DSNT [49], RCMF [70], GAT [71], XGL-T [72].”
Event extraction for visual: De-biasing with causality-guided attention mechanism
Wang, C., Yang, J., & He, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130783 · Cited by 3
SubICap: Towards Subword-informed Image Captioning
Sharif, N., Bennamoun, M., Liu, W., & Shah, S. A. A. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00358 · Cited by 4
Semantic Attribute Enriched Storytelling from a Sequence of Images
Malakan, Z. M., Hassan, G. M., Jalwana, M. A. A. K., Aafaq, N., & Mian, A. S. (2021) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta52665.2021.9647213 · Cited by 3
Show, Attend to Everything, and Tell: Image Captioning with More Thorough Image Understanding
Karimpour, Z., Sarfi, A., Asadi, N., & Ghasemian, F. (2020) · International Conference on Computer and Knowledge Engineering · doi:10.1109/iccke50421.2020.9303609 · Cited by 3
“Image captioning [13]–[15], one of the most important and challenging cross-modal tasks of machine learning, has many crucial applications, such as aiding visually impaired people and improving the medical image understanding.”
BackgroundAttention on Attention Sparse Dense Convolutional Network for Financial Signal Processing
Zhu, T., Li, J., Liu, X., Jiang, Y., & Xia, S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414397 · Cited by 3
“Attention mechanism was first used for machine translation [17] and then for computer vision [18, 19] and sequence modeling.”
BackgroundMethodRUC_AIM3 at TRECVID 2020: Ad-hoc Video Search & Video to Text Description.
Zhao, Y., Song, Y., Chen, S., & Qin, J. (2020) · TRECVID · Cited by 3
Label Importance Ranking with Entropy Variation Complex Networks for Structured Video Captioning
Tian, W., & Hu, Y. (2021) · Traitement du signal · doi:10.18280/ts.380403 · Cited by 3
“Huang et al. [20, 21] added attention mechanism to image captioning.”
Shi, Z., Liu, H., Min, M. R., Malon, C. D., Li, L. E., & Zhu, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.171 · Cited by 3
A Survey of Face Image Inpainting Based on Deep Learning
Su, S., Yang, M., He, L., Shao, X., Zuo, Y., & Qiang, Z. (2022) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-99191-3_7 · Cited by 4
Attention Is All You Need to Tell: Transformer-Based Image Captioning
Chordia, S., Pawar, Y., Kulkarni, S., Toradmal, U., & Suratkar, S. S. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-1018-0_52 · Cited by 3
Image Understanding by Captioning with Differentiable Architecture Search
Hosseini, R., & Xie, P. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548150 · Cited by 3
“Our primary baselines include: LSTM [32] and LSTM-A[37], which are non-attention based; SCST[30] that proposes employing attention over the grid of features; RFNet[14] merges CNN features by adopting recurrent fusion networks; Up-Down[2] uses attention over regions; GCN-LSTM[36] uses visual relations between image reg…”
MethodImage Captioning with Synergy-Gated Attention and Recurrent Fusion LSTM
Yang, Y., Chen, L., Pan, L., & Hu, J. (2022) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2022.10.010 · Cited by 3
Exploring adaptive attention in memory transformer applied to coherent video paragraph captioning
Cardoso, L. V., Guimarães, S. J. F., & Patrocínio, Z. K. G. (2022) · IEEE International Conference on Multimedia Big Data · doi:10.1109/bigmm55396.2022.00013 · Cited by 3
“Recently, transformers [1] have shown to be useful for many sequence-related deep learning tasks, such as machine translation [2], information retrieval [3], text classification [4], document summarization [5], image classification [6], image captioning [7], [8], video captioning [9], and others [10].”
BackgroundMethodInfluentialCITE: Compact Interactive TransformEr for Multilingual Image Captioning
Xu, Y., Hu, Z., Zhou, Y., Hao, S., & Hong, R. (2023) · International Conference on Image and Graphics Processing (ICIGP) · doi:10.1145/3582649.3582658 · Cited by 3
“The models we compare include SGAE [23], ORT [7], AoANet [8], M2 Transformer [2], and X-Transformer [17].”
BackgroundMethodInfluentialBridge the Gap: High-level Semantic Planning for Image Captioning
Yuan, C., Bai, Y., & Yuan, C. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.281 · Cited by 3
“Specifically, on CIDEr-D score our model achieves 121.8% with XE loss training and 133.4% with CIDEr-D optimization, which makes a significant improvement over the previous best model AoANet by 3.6%.”
BackgroundMethodInfluentialData augmentation to improve robustness of image captioning solutionsPreprint
Bujimalla, S., Subedar, M. M., & Tickoo, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05437 · Cited by 3
“We then train the AoANet image captioning model [4] using these features and cross-entropy loss for 25 epochs with a minibatch size of 50.”
MethodGeometry-Entangled Visual Semantic Transformer for Image CaptioningPreprint
Cheng, L., Wei, W., Zhu, F., Liu, Y., & Miao, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14137 · Cited by 3
Aminimehr, A., Khani, P., Molaei, A., Kazemeini, A., & Cambria, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4385953 · Cited by 3
Entri: Ensemble Learning with Tri-Level Representations for Explainable Scene RecognitionPreprint
Aminimehr, A., Molaei, A., & Cambria, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4482110 · Cited by 3
“Over the years, numerous studies have focused on developing image captioning systems that effectively convey visual information in a textual format [56, 57].”
BackgroundGowda, S. N., Pandey, D., & Gowda, S. N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573122 · Cited by 3
CrowdCaption++: Collective-Guided Crowd Scenes Captioning
Wang, L., Li, H., Zhang, M., Qiu, H., Meng, F., Wu, Q., & Xu, L. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3328189 · Cited by 3
Learning Multi-Scale Knowledge-Guided Features for Text-Guided Face Recognition
Hasan, M. M., Sami, S. M., Nasrabadi, N. M., & Dawson, J. M. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3466216 · Cited by 2
Advancing Causal Intervention in Image Captioning With Causal Prompt
Yu, Y., Kim, Y., & Ro, Y. M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3487200 · Cited by 2
“These compared methods are mainly classified into two categories: 1) methods that employ up–down features, including SCST [54], up–down [1], AoANet [63], vanilla transformer [64], M 2 transformer [13], X-transformer [21], DRT [65], double-stream position learning transformer network (DSPLTN) [66],…”
SeaCap: Multi-Sight Embedding and Alignment for One-Stage Image Captioner
Wang, B., Zhang, Z., Zhao, M., Jin, X., Xu, M., & Wang, M. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535303 · Cited by 2
“…categories: i) Two-stage methods, which adopt offline features directly to reason descriptions, including RFNet [66], Up-Down [6], ORT [13], AoANet [46], X-LAN [31], M 2 T [4], X-Transformer [31], DLCT [12], [67], DRT [29], RST-Net [5], S 2 Transformer [68], FutureCap [69], LightCap [70], ConCap…”
Fully Semantic Gap Recovery for End-to-End Image Captioning
Gao, J., Zhang, L., Li, J., & Mao, Z. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3558088 · Cited by 2
“E XPERIMENTS A. Dataset and Evaluation Metrics 1) Datasets: We conduct experiments on widely used image captioning dataset in previous works [1], [2], [4], [5], [6], [7], [8], [9], i.e. , MSCOCO [69].”
InfluentialEnhancing visual contextual semantic information for image captioning
Wang, R., Li, S., Xue, L., & Yang, J. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02634-9 · Cited by 2
“…model with other offline and online evaluation methods, These include SCST [38], Up-Down [3], RFNet [62], GCN-LSTM [21], SGAE [48], ORT [49], AoANet [22], M2Transformer [24], X-Transformer [23], TCIC [50], RST Net [28], S2Transformer [11], Dynamic Transformer [34], DGET [8], DSPLTN [33],…”
News Image Captioning via Separate Attention on Entity Categories
Ajankar, S., & Dutta, T. (2025) · IEEE Access · doi:10.1109/access.2025.3582655 · Cited by 2
“Further studies have explored different approaches [3], [10], [28], [29], [30], [31], [32] incorporating variations in attention mechanisms, visual and textual feature extraction techniques, language models, standard / pre-training / domain-specific datasets, etc.”
GDVT: geometrically aware dual transformer encoding visual and textual features for image captioning
Chang, Y., Ma, H.-S., Li, S.-C., Jaysawal, B. P., & Huang, J.-W. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00836-6 · Cited by 2
Enhancing image captioning with asynchronous dual attention vision transformer
Patra, B., & Kisku, D. R. (2025) · Intelligent Data Analysis · doi:10.1177/1088467x251351627 · Cited by 2
“Synchronized attention mechanisms 13,25 require the simultaneous processing of input data of different modalities, which lacks the progressive refinement of visual features.”
HIN: Hierarchical Interaction Network for Image Captioning
Song, C., Zhou, W., Jiao, H. Y., Huang, W., Li, J., & Huang, Y. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3769866 · Cited by 2
“With only 79.3M parameters, our method is 8.1M smaller than AoANet [13] (87.4M) and 59M smaller than X-Transformer [26] (138.3M), despite outperforming them by substantial margins in CIDEr (147.2 vs. 129.8/132.8).”
InfluentialDual-stage pixel transformer with enhanced visual context for image captioning
Yang, J., Liu, A., Wang, R., & Xue, L. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01965-5 · Cited by 2
“The methods we compared are listed as follows: AoANet [28], M 2 [10], X-Transformer [29], DRET [29], GET [19], TCIC [52], RSTNet [49], DLCT [24], SDATR [53], CATNet [54], S2 Transformer [55], VCT [56], RCT [57], MAENet [32], DVAT [58], DASNet [59], DRTran [60] and C-Transformer [61].”
Enhancing image captioning with spatial relational attention and grid decoder
Deng, X., Zhu, Y., & Xu, H. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02183-2 · Cited by 2
“Huang et al. [8] developed a transformer variant based on a dual attention mechanism (attention on attention, AoA), which enables a more precise focus on key visual regions, thereby improving the accuracy of generated descriptions.”
Cross-multi-modal seamless training for image captioning
Hossain, M. S., Aktar, S., Babar, A. H., Khan, M. F., Huang, X., Gu, N., & Huang, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130653 · Cited by 3
Recent Progress in Deep Learning for Chest X-Ray Report Generation
Salhi, M., & Akhloufi, M. A. (2026) · BioMedInformatics · doi:10.3390/biomedinformatics6010003 · Cited by 3
“AERMNet improves upon the AOANet baseline [65] through the addition of a dual-layer LSTM decoder, based on the Mogrifier LSTM [66].”
LrSC: A novel low-rank spatial-channel attention-driven transformer for image captioning
Hossain, M. S., Liu, X., Aktar, S., Mujahid, M., & Huang, Z. (2026) · Information Sciences · doi:10.1016/j.ins.2026.124157 · Cited by 2
Validity-Based Sampling and Smoothing Methods for Multiple Reference Image Captioning
Nagasawa, S., Watanabe, Y., & Iyatomi, H. (2021) · doi:10.18653/v1/2021.maiworkshop-1.6 · Cited by 2
“The hyper-parameter of LS was set to 0.2 according to Huang et al. (2019).”
BackgroundMethodInfluentialAttention Residual Network with 3D convolutional neural network for 3D Human Pose Estimation
Yan, J., & Mei, K. (2021) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar52367.2021.9517546 · Cited by 2
RETRACTED: Video localized caption generation framework for industrial videos
Khurana, K., & Deshpande, U. A. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-212381 · Cited by 2
“Self-attention in single LSTM-based language models was adopted in [48] by augmenting the LSTM with attention adds one more step of attention on top of visual self-attention.”
RetractedSemantic space captioner: generating image captions step by step
Zhu, C., Xia, Y., & Lu, Q. (2022) · Journal of Electronic Imaging · doi:10.1117/1.jei.31.6.063021 · Cited by 2
Text-video retrieval method based on enhanced self-attention and multi-task learning
Wu, X., Qian, J., & Wang, T. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14589-6 · Cited by 2
“Our approach modifies self-attention to get better attention results, which is inspired by attention on attention (AoA) in [13].”
MethodEfficientNet-Transformer for image captioning in Bahasa
Al-Faruq, U. A. A., & Fudholi, D. H. (2023) · AIP conference proceedings · doi:10.1063/5.0118155 · Cited by 2
Gated Cross Word-Visual Attention-Driven Generative Adversarial Networks for Text-to-Image Synthesis
Lai, B., Ma, L., & Tian, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26293-7_6 · Cited by 2
“The AoA method [4] is also applied to obtain the V2W attentional information.”
MethodIntegration of the latent variable knowledge into deep image captioning with Bayesian modeling
Barati, A., Farsi, H., & Mohamadzadeh, S. (2023) · IET Image Processing · doi:10.1049/ipr2.12790 · Cited by 2
SERG: A Sequence-to-Sequence Model for Chinese ECG Report Generation
Yang, C., Li, Z., Fan, H., Chen, Y., Lin, N., & Wei, N. (2022) · International Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI) · doi:10.1145/3584376.3584501 · Cited by 2
“Existing sequence-to-sequence models have been used in various applications, such as translation [6], [7], image captioning [8], [9], and etc.”
MethodFashion item captioning via grid-relation self-attention and gated-enhanced decoder
Tang, Y., Zhang, L., & Yuan, Y. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15492-w · Cited by 3
“6 Performance changes of AOA [12] on FACAD test set Besides, we also present the comparison experiments between fashion item captioning and fashion image captioning on the FACAD dataset.”
InfluentialLâm, K. N., Le, D.-K. T., Ngo, T. D., & Kalita, J. K. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-35641-4_9 · Cited by 2
A Context Semantic Auxiliary Network for Image Captioning
Li, J., & Shao, X. (2023) · Information · doi:10.3390/info14070419 · Cited by 2
Few-Shot Personality-Specific Image Captioning via Meta-Learning
Hosseinzadeh, M., & Wang, Y. (2023) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv60082.2023.00048 · Cited by 2
“Image captioning [28], [40], [3], [42], [4], [21], [9], [17], [39], [29], [18] has become a popular research topic in recent years.”
BackgroundContour-Augmented Concept Prediction Network for Image Captioning
Wang, T., Chen, W., Li, J., Peng, Y., & Mao, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44210-0_15 · Cited by 2
Robust Image Captioning Using Knowledge Distillation
Guo, J. (2023) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece58870.2023.10263327 · Cited by 2
CroMIC-QA: The Cross-Modal Information Complementation Based Question Answering
湯浅, 順., Liu, B., Sun, C., Xu, Z. X., Ma, L., & Wang, B. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3326616 · Cited by 3
“To improve the performance, the attention mechanism and object detection approaches are incorporated [3], [9], [35], [36], [37], [38], [39], [41], [45], and meanwhile, transformer-based architectures are also utilized to enhance the language model [42], [43], [44].”
MethodThe Impact of Text Generation Techniques on Neural Image Captioning: An Empirical Study
Ding, L., Jiang, M., Nie, L., Qing, Z., & Ding, Z. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01410114 · Cited by 2
“Image caption : Image captioning [29], [30], [31], [12] achieves significant improvements over the neural encoder-decoder framework [6].”
BackgroundMethodInfluentialRajagopal, A. R., Nirmala, V., & Arun, M. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-3481-2_53 · Cited by 2
Deep Vision Transformer and T5-Based for Image Captioning
Lâm, K. N., Nguyen, H. T., Phuoc, V., & Kalita, J. K. (2023) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf60135.2023.10471815 · Cited by 2
“The OSCAR model outperforms other image captioning models [10], [11].”
MethodInfluentialCaptionCraft: VGG with LSTM for Image Insights
Syed Abudhagir, U., Vignesh, K., Harish, U., Avinash, V., & Subbarao, M. V. (2024) · International Conference on Trends in Engineering Systems and Technologies (ICTEST) · doi:10.1109/ictest60614.2024.10576172 · Cited by 2
Attention on Attention as a part of decoder for the vehicle license plate recognition
Zhumabay, Y. (2024) · IEEE AITU: Digital Generation · doi:10.1109/ieeeconf61558.2024.10585566 · Cited by 2
“In the second architecture Figure 3 the Attention on attention layer comes from the paper [37] and the paper inspires the decoder block an extended attention mechanism for scene text recognition [35], which outperformed previous approaches by all benchmarks.”
BackgroundMethodEnhancing Image Captioning with Transformer-Based Two-Pass Decoding Framework
Su, J., Mou, Y., & Xie, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5663-6_15 · Cited by 2
Spatial guided image captioning: Guiding attention with object's spatial interaction
Du, R., Zhang, W., Li, S., Chen, J., & Guo, Z. M. (2024) · IET Image Processing · doi:10.1049/ipr2.13124 · Cited by 2
Self-citationMemory positional encoding for image captioning
Yang, X., He, S., Zhang, J., Ma, S., Hou, Z., & Sun, W. (2024) · Signal Processing Image Communication · doi:10.1016/j.image.2024.117201 · Cited by 2
Meshed Context-Aware Beam Search for Image Captioning
Zhao, F., Yu, Z., Wang, T., & Zhao, H. (2024) · Entropy · doi:10.3390/e26100866 · Cited by 2
“The attention on attention (AoA) [43] model improves the focus on important image areas by refining the attention mechanism, while the dual-level collaborative transformer (DLCT) [20] employs two levels of attention to balance global and local details in captions.”
BackgroundBidirectional interactive alignment network for image captioning
Cao, X., Yan, P., Hu, R., & Li, Z. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01559-7 · Cited by 2
“Huang et al. [24] introduced the “Attention on Attention” (AoA) module to focus on the relationship between Q and V, avoiding misleading information due to the irrelevance between Q and K/V. Cornia et al. [4] considered the visual relationship between low-level and high-level features and encoded…”
Cao, Z., Xia, J., Shi, Y., & Zhou, M. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802814 · Cited by 2
“We conduct performance comparisons between LDTNet and its competitive peers, i.e. Up-Down [8], SGAE [32], AoANet [24], SMArT [4], M 2 Transformer [33], SCD-Net [34], X-Transformer [7], DLCT [14], and RSTNet [2].”
Combining Region-Guided Attention and Attribute Prediction for Thangka Image Captioning Method
Zhang, F., Kang, W., & Hu, W. (2025) · IEEE Access · doi:10.1109/access.2025.3526954 · Cited by 2
Cao, Z., Xia, J., & Zhou, M. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3553260 · Cited by 3
“…(GCN-LSTM) [52], scene graph auto-encoder (SGAE) [53], shallow memory-aware transformers (SMArTs) [36], attention on attention network (AoANet) [54], Meshed Transformer with memory ( M 2 Transformer) [15], semantic-conditional diffusion networks (SCD-Net) [47], X-linear attention trans-former…”
Making Images Speak: Human-Inspired Image Description Generation
Sebbane, C., Belhajem, I., & Rziza, M. (2025) · Information · doi:10.3390/info16050356 · Cited by 2
“Extensions such as AoA [26] improved spatial alignment.”
Injecting object pose relationships into image captioning via attention capsule networks
Yu, H., Li, H., Li, H., Han, X., & Liu, H. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113310 · Cited by 2
IMGEF: integrated multimodal graph-enhanced framework for radiology report generation
Usman, M., Hou, X., Guo, Y., Liang, Z., & Yijia, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01858-7 · Cited by 2
“Proposed an attention mechanism to evaluate the relationship between query inputs and attention outputs for generating captions [12].”
A Novel Transformer-Based Framework for Chest X-Ray Captioning with Clinical Sign Detection
Tran, T. T., Nguyen, D. V. M., & Huynh, H. T. (2025) · SN Computer Science · doi:10.1007/s42979-025-04032-y · Cited by 2
Dual-level semantic collaboration and inference network for medical image report generation
Zhang, J., Liu, Y., Shao, M., Chen, C., Wang, Z., Li, J., Wan, Y., & Yu, P. S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114278 · Cited by 2
Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generation
Tian, Y., Mao, L., & Song, Y. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356692 · Cited by 2
“Considering attention mechanism is demonstrated to be an effective approach to identify important content [21], [34], [35], [36], [37].”
Weakly Supervised Content Selection for Improved Image Captioning
Chandu, K. R., Sharma, P. K., Changpinyo, S., Thapliyal, A., & Soricut, R. (2020) · Cited by 2
Joint Common Sense and Relation Reasoning for Dense Relational Captioning
Cao, S., Liu, W., An, G., & Ruan, Q. (2020) · International Conference on Signal Processing · doi:10.1109/icsp48669.2020.9321009 · Cited by 2
An Object-Extensible Training Framework for Image Captioning
Wu, Y., Zhang, Y., & Yuan, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_64 · Cited by 2
Transformer with Prior Language Knowledge for Image Captioning
Yan, D., Yu, W., Zhang, Z., & Gong, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92270-2_4 · Cited by 2
Hybrid Attention Distribution and Factorized Embedding Matrix in Image Captioning
Wang, J., & Feng, J. (2020) · IEEE Access · doi:10.1109/access.2020.3018546 · Cited by 3
“…GCN-LSTM [36], which predicts visual relationships between every two entities in the image and encodes the relationship information into feature vectors; and finally AoANet [14] uses an information vector and an attention gate to obtain attended information based on transformer language model.”
MethodResultInfluentialTowards Multilingual Image Captioning Models that Can Read
García, R., Beltrán, B., Hernández, C., & Ayala, D. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89820-5_2 · Cited by 2
Fine-Grained Cross-Modal Retrieval with Triple-Streamed Memory Fusion Transformer Encoder
Guo, W., Huang, H., Kong, X., & He, R. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859738 · Cited by 2
“Among the various multi-modal research topics [1,2], cross-modal retrieval is one of the most fundamental and wildly studied tasks.”
BackgroundFeature-Fusion Parallel Decoding Transformer for Image Captioning
Zhu, C., Xia, Y., & Lu, Q. (2022) · Academic Journal of Science and Technology · doi:10.54097/ajst.v2i1.905 · Cited by 2
“We report the performance of the FPDT model with models that have achieved cutting-edge results in recent years in the same experimental settings, including SCST [16], Up-Down [1], RFNet [17], GCN-LSTM [18], SGAE [19], ORT [20], AoANet [21], 𝑀 Transformer [23], X-Transformer [22].”
MethodImproved Transformer with Parallel Encoders for Image Captioning
Lou, L., Lu, K., & Xue, J. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956518 · Cited by 2
“Huang et al. [4] proposed an attention on attention (AoANet) module to extend the conventional attention mechanism to determine the relevance between attention results and queries.”
BackgroundMethodInfluentialDual-stream Self-attention Network for Image Captioning
Wan, B., Jiang, W., Fang, Y., Wen, W., & Liu, H. (2022) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip56404.2022.10008904 · Cited by 2
“III shows the comparison of our method against existing SOTA ICMs including SCST [30], Up-Down [1], RFNet [18], GCN-LSTM [45], SGAE [44], AoANet [15], ORT [12], Transformer [37], M(2)Transformer [4], XTransformer [27], RSTNet [48] and SDATR [25] on the MSCOCO Karpathy test split.”
BackgroundMethodInfluentialTraVL: Transferring Pre-trained Visual-Linguistic Models for Cross-Lingual Image Captioning
Zhang, Z., Lu, P., Jiang, D., & Chen, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25198-6_26 · Cited by 2
Survey on the Research Progress and Development Trend of Vision-and-Language Navigation
Niu, K., & Wang, P. (2022) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2022.19249 · Cited by 2
TB-Transformer: Integrating Mouse Trace with Object Bounding-box for Image Caption
Lin, N., Guo, S., & Xie, L. (2022) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys57074.2022.00133 · Cited by 2
Preference-Controlled Multi-Objective Reinforcement Learning for Conditional Text Generation
Chen, W., Tian, J., Fan, C., Li, Y., He, H., & Jin, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26490 · Cited by 2
“The first is a commonly used strategy in previous work (Fu, Feng, and Cunningham 2019; Huang et al. 2019; Ruan, Ling, and Zhu 2020), which keeps the top-1 text among all ranked texts through beam search.”
BackgroundMethodInfluentialTowards Few-shot Image Captioning with Cycle-based Compositional Semantic Enhancement Framework
Zhang, P., Bai, Y., Su, J., Huang, Y., & Long, Y. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191558 · Cited by 2
“Further, our framework compares with AoANet [35], ORT [34] and M 2 Transformer [13], which apply Transformer for encoding image regions.”
MethodImproving Visual Question Answering by Multimodal Gate Fusion Network
Xiang, S., Chen, Q., Fang, X., & Guo, M.-H. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191453 · Cited by 2
“To make the network smaller, we make dh = d/h. 2) Attention-on-Attention Mechanism: The AoA module was first proposed by Huang [3].”
BackgroundMethodInfluentialTraffic Scene Captioning with Multi-Stage Feature Enhancement
Zhang, D., Ma, Y., Liu, Q., Wang, H., Ren, A., & Liang, J. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.038264 · Cited by 2
Improving fashion captioning via attribute-based alignment and multi-level language model
Tang, Y., Zhang, L., Yuan, Y., & Chen, Z. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05167-2 · Cited by 2
Aminimehr, A., Khani, P., Molaei, A., Kazemeini, A., & Cambria, E. (2024) · doi:10.1145/3665601.3669841 · Cited by 2
“Many research efforts have been put into developing an image captioning system capable of explaining visual data descriptively in a text-based format [46, 47, 48].”
BackgroundLearn and Tell: Learning Priors for Image Caption Generation
Liu, P., Peng, D., & Zhang, M. (2020) · Applied Sciences · doi:10.3390/app10196942 · Cited by 2
“(2) Involving extra clues in attention weights or not—for example, studies [11,24,25] obtained their attention weights only from previous attention calculations, while the authors of [18] combined geometry clues with previously calculated weights into final attention weights.”
BackgroundInfluentialContext-Fused Guidance for Image Captioning Using Sequence-Level Training
Feng, J., & Zhao, J. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9743123 · Cited by 2
“In [5], the attention-on-attention module employs an attention gate to transform the result from a standard attention mechanism.”
Background
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
