The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Cited by
Sorted by citations
Items 601–700 of 1,002
Zero-TextCap: Zero-shot Framework for Text-based Image Captioning
Xu, D., Zhao, W., Cai, Y., & Huang, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612571 · Cited by 5
“Up-Down [2] and AOA [13] are popular image captioning models that do not incorporate OCR tokens in their training and inference process.”
BackgroundCONICA: A Contrastive Image Captioning Framework with Robust Similarity Learning
Deng, L., Zhong, Y., Wang, M., & Zhang, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612326 · Cited by 5
“For instance, AoANet [12] builds self/cross-attention between vision and language, M 2 transformer [7] interacts multi-level vision and language features, and COSNet utilizes a semantic ranker [19] to filter out the irrelevant semantic while inferring the missing relevant language semantic visually…”
BackgroundMethodAttention Enhanced Network with Semantic Inspector for Medical Image Report Generation
Lin, Y., Tang, Q. J., Wang, H., Huang, C., Ekong, F., Wang, X., Xiao, F., & Yu, Y. (2023) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai59109.2023.00043 · Cited by 5
“Medical report generation derives from a more general task, image caption, whose models are mainly based on encoder-decoder structure[1, 2, 3, 4, 5].”
BackgroundAdvancing image captioning with V16HP1365 encoder and dual self-attention network
Jaiswal, T., Pandey, M., & Tripathi, P. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18467-7 · Cited by 5
“The Attention-on-Attention model was implemented by Huang et al. [18] to focus on specific semantic concepts in images by using semantic attention.”
GRPIC: an end-to-end image captioning model using three visual features
Peng, S., Xiong, C., Liu, L., Yang, L. T., & Chen, J. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02352-8 · Cited by 5
“Early image captioning networks are always based on CNN+RNN models [1, 2, 12, 13], where most encoders of models employed pre-trained CNNs (such as ResNet, VGG, Inception) to transform images into fixed length grid-level features.”
Self-citationReverse Stable Diffusion: What prompt was used to generate this image?
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. A. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104210 · Cited by 9
Semantic‐meshed and content‐guided transformer for image captioning
Li, X., Zhang, W., Sun, X., & Gao, X. (2022) · IET Computer Vision · doi:10.1049/cvi2.12099 · Cited by 5
Bangla Image Captioning With Bidirectional GRU & Attention Mechanism
Zohora, F. T., & Abedin, Z. (2022) · International Conference on Innovations in Science, Engineering and Technology (ICISET) · doi:10.1109/iciset54810.2022.9775869 · Cited by 5
The Role of Transformer-based Image Captioning for Indoor Environment Visual Understanding
Fudholi, D. H., & Nayoan, R. A. N. (2022) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/120138 · Cited by 7
“There are other methods that also utilized attention mechanism to their encoder-decoder methods, such as Attention on Attention (AoA) [24], Auto-Encoder Scene Graph (SGAE) [25], Adaptive attention via visual sentinel [26], gradient policy optimization of SPIDEr [27], and Recurrent Fusion Network (RFNet) [28].”
MethodLG-MLFormer: local and global MLP for image captioning
Jiang, Z., Wang, X., Zhai, Z., & Cheng, B. H. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00266-9 · Cited by 5
“To prove that the excellent performance of our model does not depend entirely on the visual features extracted by ResNext101, we also conduct experiments to compare our proposed LG-MLFormer with the SOTAs on the two sources”
MethodSemantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning
Xiang, N., Chen, L., Liang, L., Rao, X., & Gong, Z. (2023) · Electronics · doi:10.3390/electronics12173549 · Cited by 7
Exploring Spatial-Based Position Encoding for Image Captioning
Yang, X., He, S., Wu, J., Yang, Y., Hou, Z., & Ma, S. (2023) · Mathematics · doi:10.3390/math11214550 · Cited by 5
XlanV Model with Adaptively Multi-Modality Feature Fusing for Video Captioning
Huang, Y., Cai, Q., Xu, S., & Chen, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3416290 · Cited by 5
“Most existing attention mechanisms like soft-attention [26], scaled dotproduct attention [20], and attention on attention [8] mainly focus on modeling the interaction between the query and key in the 1 order.”
BackgroundMethodSelf-citationWithout detection: Two‐step clustering features with local–global attention for image captioning
Li, X., Zhang, W., Sun, X., & Gao, X. (2022) · IET Computer Vision · doi:10.1049/cvi2.12087 · Cited by 5
Show and Speak: Directly Synthesize Spoken Description of Images
Wang, X., Feng, S., Zhu, J., Hasegawa‐Johnson, M., & Scharenborg, O. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414021 · Cited by 6
“The transcribed textual captions are then evaluated using evaluation metrics for image captioning [7, 8]: BLEU1 (B1), BLEU2 (B2), BLEU3 (B3), BLEU4 (B4), METEOR (M), ROUGE (R), and CIDEr (C).”
BackgroundMethodCausal Attention for Vision-Language TasksPreprint
Yang, X., Zhang, H., Qi, G.-J., & Cai, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03493 · Cited by 5
A Survey on Natural Language Video LocalizationPreprint
Liu, X., Nie, X., Tan, Z., Guo, J. J., & Yin, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00234 · Cited by 6
“In addition, they used information gate [48] to construct Fine-grained Iterative Attention Net-work (FIAN) so that the information from each modal can guide the other modality iteratively.”
Li, W., Li, X., Li, Z., Yu, J., & Chen, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130622 · Cited by 4
Dual dynamic transformer for image captioning
Shan, C., Song, C., Zou, T., Li, J., & Liu, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128597 · Cited by 4
Multimodal learning with feature fusion transformer for image captioning
Zhu, W., & Yuan, F. (2025) · Displays · doi:10.1016/j.displa.2025.103126 · Cited by 5
Visual spatial relationship sensitive transformer for image captioning
Piao, X., Jin, D., Kwon, M. J., & Gu, Y. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-28290-1 · Cited by 4
MSSA: memory-driven and simplified scaled attention for enhanced image captioning
Hossain, M. A., Ye, Z., Hossen, M. B., Rahman, M. A., Islam, M. S., & Abdullah, M. I. (2026) · Scientific Reports · doi:10.1038/s41598-026-40164-8 · Cited by 4
Incorporating semantic consistency for improved semi-supervised image captioning
Wu, B., & Wo, Y. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17577-y · Cited by 4
“Consistent with existing methods under this setting, we adopt AoANet [36] as the Base model.”
Region-Focused Network for Dense Captioning
Huang, Q., Li, P., Huang, Y., Shuang, F., & Cai, Y. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3648370 · Cited by 4
“With the remarkable success in various tasks, recent studies [4, 9, 13, 22, 33, 47, 54, 58] try to introduce Transformer [3, 34, 39] architecture into image captioning.”
BackgroundMethodYuan, B., Zhao, D., Liu, Z., Li, W., & Li, T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680654 · Cited by 4
“Attention-basedmeth-ods[1,23]introducesemanticattentionmechanismstomodel contextualinformationamongareasinanimage.”
RNIC-A retrospect network for image captioning
Yi, X., Hua, R., Fu, Y., Zheng, D., & Wang, Z. (2022) · Soft Computing · doi:10.1007/s00500-021-06622-3 · Cited by 4
“The models include: Stack-VS Attention[30], which proposes a visual-semantic attention based multi-stage framework; GCN-LSTM[16], which explores visual relationship for boosting image captioning; LBPF[31], which can embed previous visual information and look into future; SGAE[32], which introduce auto-encoding scene g…”
BackgroundSituational Perception Guided Image Matting
Xu, B., Xie, J., Huang, H. P., Li, Z., Lu, C., Tang, Y., & Guo, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548036 · Cited by 4
“[24] propose an attention model that first generates an information vector and an attention gate, and then adds secondary attention using element-wise multiplication to aggregate the attended features.”
BackgroundVisual lifelog retrieval: humans and machines interpretation on first-person images
Yen, A.-Z., Fu, M.-H., Ang, W.-H., Chu, T.-T., Tsai, S.-H., Huang, H., & Chen, H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14344-x · Cited by 4
“Recently, a lot of works [1, 10, 18, 34] have worked on image caption generation by utilizing large-scaled datasets collected from the web, such as MS-COCO [28] and ImageNet [21].”
BackgroundSequential Vision to Language as Story: A Storytelling Dataset and Benchmarking
Malakan, Z. M., Anwar, S., Hassan, G. M., & Mian, A. S. (2023) · IEEE Access · doi:10.1109/access.2023.3293646 · Cited by 4
“can be classified further into rule-based methods [16] and deep learning-based methods [17], [18].”
BackgroundBounding and Filling: A Fast and Flexible Framework for Image Captioning
Ma, Z., Wang, C., Huang, B., Zhu, Z., & Zhang, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_37 · Cited by 4
“However, many high-performing captioning models utilizing an autoregressive manner [14,18,24] generate descriptions incrementally, word by word, resulting in significant inference latency.”
BackgroundImproving Image Captioning through Visual and Semantic Mutual Promotion
Zhang, J., Xie, Y., & Liu, X. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612480 · Cited by 5
“Huang et al. [17] propose an attention on attention module, which further determines the relevance of attention results through a gating mechanism after multi-head attention.”
BackgroundPSNet: position-shift alignment network for image caption
Xue, L., Zhang, A., Wang, R., & Yang, J. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00307-3 · Cited by 4
“…the other methods both on offline and online evaluation, including SCST [39], Up-Down [6], RFNet [40], GCN-LSTM [41], SGAE [42], ORT [26], AoANet [43], M 2 Transformer [12], X-Transformer [44], TCIC [45], RSTNet [13], S 2 Transformer [14],Dynamic Transformer [9],DSPLTN [46],LG-MLFormer [47],RCT…”
MethodBERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings
Hosseini, M., Munia, M. S., & Khan, L. R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1030 · Cited by 6
“…other NLP tasks (e.g., punctuation insertion (Hosseini and Sameti, 2017) and question answering (Qu et al., 2019)) and also utilize it in other domains where deep learning models are used (e.g., biosignal analysis (Munia et al., 2020, Munia et al., 2023) and image captioning (Huang et al., 2019)).”
MethodHafeth, D. A., Lal, G. R., Al-Khafajiy, M., Baker, T., & Kollias, S. (2023) · International Conference on Developments in eSystems Engineering (DeSE) · doi:10.1109/dese60595.2023.10468995 · Cited by 5
“When applied to image captioning, the attention mechanism helps the model focus on specific image regions or features while generating a textual description [16].”
MethodClustering-based mask recovery for image captioning
Liang, 徐. 良. X., Li, C., & Tian, L. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128127 · Cited by 4
LLAFN-Generator: Learnable linear-attention with fast-normalization for large-scale image captioning
Yang, X., Tian, X., Wu, J., Yang, X., Ma, S., Qi, X., & Hou, Z. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104088 · Cited by 4
Sentinel mechanism for visual semantic graph-based image captioning
Xiao, F., Zhang, N., Xue, W., & Gao, X. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109626 · Cited by 4
CLIP-Driven Low-Cost Image Captioning
Wang, Y., Xu, J., & Sun, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650602 · Cited by 4
“The compared approaches include: UpDown [2], LBPF [37], AOANet [19], M 2 Trans-former [7], and DLCT [31]; ClipCap [32] and CLIP-M 2 [5].”
BackgroundMethodImage restoration in frequency space using complex-valued CNNs
Shah, Z. H., Muller, M. W., Hübner, W., Ortkrass, H., Hammer, B., Huser, T. R., & Schenck, W. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1353873 · Cited by 4
“…gates (AGs) are widely used attention mechanisms in the field of natural language processing (NLP) (Li et al., FrontiersinArtificialIntelligence frontiersin.org 2018) and computer vision for segmentation (Zuo et al., 2021), image captioning (Huang et al., 2019), and classification (Wang et al., 2017).”
A Transformer-Based Model for Image Caption Generation with Memory Enhancement
Ome, S. A., Azhar, T., & Asaduzzaman, A. (2024) · IEEE International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE) · doi:10.1109/wiecon-ece64149.2024.10915030 · Cited by 4
“On top of the multi-head attention mechanism, AoANet [14] improves performance by adding a gated linear layer while maintaining the original transformer architecture.”
Attribute-Based Learning for Remote Sensing Image Captioning in Unseen Scenes
Guo, Z., Liu, H., Ren, Z., Jiao, L., Gou, S., & Li, R. (2025) · Remote Sensing · doi:10.3390/rs17071237 · Cited by 5
Dual-visual collaborative enhanced transformer for image captioning
Mou, Z., Song, T., & Luo, H. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01775-9 · Cited by 4
“AoA [28], employing the Self Attention (SA) technique, efficiently synthesizes comprehensive representation of interactive zones within an image.”
Video captioning via a symmetric bidirectional decoder
Qi, S., & Yang, L. (2021) · IET Computer Vision · doi:10.1049/cvi2.12043 · Cited by 4
“Most dominant video captioning methods adopt the classical attentional encoder–decoder framework [4–13], which has brought significant performance improvements in neural machine translation (NMT) [14, 15] and image captioning tasks [16, 17].”
BackgroundWhat should we pay attention to when classifying violent videos?
Teixeira, M. V. A., & Avila, S. (2021) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3465481.3470059 · Cited by 4
“using the attention idea for machine translation [55], machine reading [10], image captioning [27], action recognition [34] and speech recognition [60].”
MethodA Deep Learning Approach For Bangla Image Captioning System
Kamruzzaman, T., Kamruzzaman, S., & Zaman, A. (2021) · International Conference on Informatics, Electronics & Vision (ICIEV) · doi:10.1109/icievicivpr52578.2021.9564129 · Cited by 4
Learning cross-modality features for image caption generation
Zeng, C., & Kwong, S. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01506-w · Cited by 4
“[17] proposed the Attention on Attention model inspired by the principle of multi-head attention in the transformer language model.”
MethodImprove Image Captioning by Modeling Dynamic Scene Graph Extension
Geng, M., & Zhao, Q. (2022) · International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531401 · Cited by 4
An Image Captioning Algorithm Based on Combination Attention Mechanism
Liu, J., Cheng, K., Jin, H., & Wu, Z. (2022) · Electronics · doi:10.3390/electronics11091397 · Cited by 5
Input enhanced asymmetric transformer for image captioning
Zhu, C., Xia, Y., & Lu, Q. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-022-02350-9 · Cited by 4
“Following a standard practice of image captioning [19, 22], we first optimize ourmodel with the cross-entropy loss (XE):”
MethodGenerating Spatial-aware Captions for TextCaps
Wang, Y., & Shang, L. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956709 · Cited by 4
“We evaluate the performance of our model on the TextCaps validation set and test set, and compare our results with baseline models BUTD [20] and AoANet [21] and best to date models M4C-Captioner [4], MMA-SR [5] and CNMT [6], as summarized in Table I and Table II.”
MethodDeep Region Adaptive Denoising for Texture Enhancement
Lee, S.-E., Woo, S.-M., Kim, J., Ryu, J.-H., & Kim, J. (2022) · IEEE Access · doi:10.1109/access.2022.3222826 · Cited by 4
“It has been shown great performance in many applications, including image classification [32], action recognition [33], segmentation [34], [35], image captioning [36], [37], etc. Recently, attention methods are also used for diverse feature fusion tasks [10], [12], [38], [39], [40].”
MethodContext-aware and co-attention network based image captioning model
Sharma, H., & Srivastava, S. (2023) · The Imaging Science Journal · doi:10.1080/13682199.2023.2179992 · Cited by 4
“The existing works explored the visual relationship for image cap-tioning using encoder-decoder systems [15 – 18], attention-based models [19 – 24], and relation-aware models [25 – 27].”
BackgroundMethodGAF-Net: Global view guided attribute fusion network for remote sensing image captioning
Peng, Y., Jia, Y., Chen, J., & Ji, X. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16421-7 · Cited by 4
“Later on, many image captioning models were derived [2, 5, 16, 26] and achieved good results.”
Self-citationRelation extraction in Chinese using attention-based bidirectional long short-term memory networks
Zhang, Y. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1509 · Cited by 4
“Recently, attention-based neural networks have demonstrated remarkable achievements across various tasks, such as question answering ( Lu et al., 2016 ), machine translation ( Luong, Pham & Manning, 2015 ), speech recognition ( Chorowski et al., 2015 ), and image captioning ( Huang et al., 2019 ).”
Fully-attentive iterative networks for region-based controllable image and video captioning
Cornia, M., Baraldi, L., Tal, A., & Cucchiara, R. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103857 · Cited by 5
Event-Aware Retrospective Learning for Knowledge-Based Image Captioning
Liu, A.-A., Zhai, Y., Xu, N., Tian, H., Nie, W., & Zhang, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3327537 · Cited by 4
Multimodal encoders and decoders with gate attention for visual question answering
Li, H., & Han, D. (2021) · Computer Science and Information Systems · doi:10.2298/csis201120032l · Cited by 4
“The gate attention learns from the AoA model proposed in [22], which take the results of multi-head attention calculations and queries as input.”
MethodBe Specific, Be Clear: Bridging Machine and Human Captions by Scene-Guided Transformer
Huang, Y., Zeng, Z., & Lu, Y. (2021) · doi:10.1145/3463945.3469054 · Cited by 4
“In the cross-entropy loss training stage, SG-Transformer outperforms all Transformer-based models and achieves comparable results with the state-of-the-art RNN-based model AoANet[22] in most metrics.”
BackgroundResultVariational joint self‐attention for image captioning
Shao, X., Xiang, Z., Li, Y., & Zhang, M. (2022) · IET Image Processing · doi:10.1049/ipr2.12470 · Cited by 4
Relational Graph Reasoning Transformer for Image Captioning
Xiao, X., Sun, Z., Li, T., & Yu, Y. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859926 · Cited by 4
“Currently popular Transformer [3] based models like applying self-attention [4], extending attention operator [5] or stacking multi-layer [6] can efficiently handle the problem of set and sequence modeling.”
MethodRotation Axis Focused Attention Network (RAFA-Net) for Estimating Head Pose
Behera, A., Wharton, Z., Hewage, P., & Kumar, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69541-5_14 · Cited by 4
Pixels to Prose: Understanding the Art of Image CaptioningPreprint
Sharma, A., Singh, H., & Pant, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351410 · Cited by 4
Region-guided transformer for remote sensing image captioning
Zhao, K., & Xiong, W. (2024) · International Journal of Digital Earth · doi:10.1080/17538947.2024.2400988 · Cited by 3
“Inspired by L. Huang et al. (2019), the global features are used as queries; to highlight region-related environment information, the correlation factors between region features and global features are used as values to obtain the total environment features: where b e represents the bias matrix…”
Saraf, G., Kulkarni, M. S., Sabane, R., Oswal, D., & Karnavat, R. (2024) · Asia Pacific Conference on Innovation in Technology (APCIT) · doi:10.1109/apcit62007.2024.10673555 · Cited by 3
“In the paper [28], the authors demonstrate the effectiveness of the Attention on Attention (AoA) mechanism in image cap-tioning.”
BackgroundMSP: Multimodal Self-Attention Prompt Learning
Lai, X., Ke, X., Xu, H., Wu, S., & Guo, W. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3607613 · Cited by 3
Semantics-aware transformer for 3D reconstruction from binocular images
Jia, X., Yang, S., & Guan, D. (2022) · Optoelectronics Letters · doi:10.1007/s11801-022-2055-0 · Cited by 3
“The feature extraction module employs MHA to obtain an elementary contextual feature and then introduces the attention on attention (AoA) model [17] to refine the elementary contextual feature.”
MethodObject semantic analysis for image captioning
Du, S., Zhu, H., Lin, G., Wang, D. Y., Shi, J., & Wang, J. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14596-7 · Cited by 3
“To evaluate the effect of applying our model to different language generators, we adopt Bottom-up [2] instead of AOANet [19] to generate sentences.”
MethodInfluentialGroup-Based Distinctive Image Captioning with Memory Difference Encoding and Attention
Wang, J., Xu, W., Wang, Q., & Chan, A. B. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02220-6 · Cited by 3
“We compare our model with two groups of state-of-the-art models: a) FC (Rennie et al., 2017), Att2in (Rennie et al., 2017), UpDown (Anderson et al., 2018) and AoANet (Huang et al., 2019) that aim to generate cap-tions with high accuracy; b) DiscCap (Luo et al., 2018), CL-Cap (Dai and Lin, 2017), and CIDErBtwCap (Wang…”
InfluentialGlobal information regulation network for multimodal sentiment analysis
Xie, S., Chen, Q., Fang, X., & Sun, Q. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105297 · Cited by 4
EdgeScan for IoT Contextual Understanding With Edge Computing and Image Captioning
Hafeth, D. A., Al-Khafajiy, M., & Kollias, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3492066 · Cited by 3
“To generate meaningful captions, many early approaches and techniques have achieved success through the designing encoder and decoder framework and incorporating attention mechanisms and transformer models [11], [16], [17], [18], [19]. c (cid:2) 2024 The Authors.”
Fluent and Accurate Image Captioning with a Self-trained Reward Model
Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78456-9_14 · Cited by 4
Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation
Yang, Y., Xi, W., Zhou, L., & Tang, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3518759 · Cited by 4
“Note that the ranking is computed using text similarity scores (i.e., ROUGE-L [57]) between a sentence and the sentences associated with a certain image.”
Multi-Modal Social Media Analytics: A Sentiment Perception-Driven Framework in Nanjing Districts
Xia, M., Lu, Z., & Wang, F. (2025) · IEEE Access · doi:10.1109/access.2025.3531769 · Cited by 4
Bidirectional Semantic Consistency Guided Contrastive Embedding for Generative Zero-Shot Learning
Hou, Z., Li, J., & Li, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107423 · Cited by 3
BMFNet: Bidirectional Multimodal Fusion Network for image captioning
Xue, L., Jin, Z., Wang, R., & Yang, J. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01801-w · Cited by 3
“The comparison includes models leveraging only grid features, such as SAT [3], SCST [35], RSTNet [44], and DTNet [45], as well as those exclusively based on region features, including Up-Down [46], AoA [34], M2 Transformer [8], X-Transformer [47], CIIC g [36], and DSPLTN [21].”
NuCap: A Numerically Aware Captioning Framework for Improved Numerical Reasoning
Jeong, Y., & Choi, Y. S. (2025) · Applied Sciences · doi:10.3390/app15105608 · Cited by 3
PKE-UIC: A Prior Knowledge-Enhanced Underwater Image Captioning Model
Li, H., Zhang, W., Li, Z., & Ren, P. (2024) · International Conference on Sensors, Signal and Image Processing (SSIP) · doi:10.1145/3725949.3725952 · Cited by 3
Weakly-Supervised 3D Visual Grounding Based on Visual Language Alignment
Xu, X., Yuan, Y., Zhang, Q., Wu, W., Jie, Z., Ma, L., & Wang, X. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3599032 · Cited by 3
“…pairs for model training, VLMs are able to establish precise semantic matching relationships between natural languages and 2D images, and have achieved good results in various downstream tasks such as image classification [20], [21], visual question answering [22], and image captioning [23].”
Neighbours Matter: Image Captioning with Similar Images
Wang, Q., Wang, J., Chan, A. B., Huang, S., Xiong, H., Li, X., & Dou, D. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.90 · Cited by 3
Deep Gaussian Denoiser Epistemic Uncertainty and Decoupled Dual-Attention Fusion
Ma, X., Lin, X., Helou, M. E., & Susstrunk, S. E. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506653 · Cited by 3
Spatial-Prior-Guided Attention for Small Object Detection in Overhead Catenary System
Gong, H., Zheng, S., Wu, Z., Xu, Y., Wei, Z., Xia, L., Xu, W. D., Liu, J., & Ding, D. (2021) · Journal of Circuits Systems and Computers · doi:10.1142/s021812662250075x · Cited by 3
Visual-Aware Attention Dual-Stream Decoder for Video Captioning
Sun, Z., Chen, S., & Zhong, L. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859743 · Cited by 3
“Inspired by the development of neural machine translation, the attention mechanism [1], [2], [11]–[13] has been widely used in current encoder-decoder frameworks for visual cap-tioning, which guides the decoding process by generating a weighted average over the extracted feature vectors for each…”
BackgroundMethodAn Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
Raymond, O. I., Yang, Z., & Long, J. (2021) · Applied Sciences · doi:10.3390/app112411635 · Cited by 4
Ahmed, B. H. A., & Saad, M. (2021) · Palestinian International Conference on Information and Communication Technology (PICICT) · doi:10.1109/picict53635.2021.00017 · Cited by 4
Backpropagation-Based Decoding for Multimodal Machine Translation
Yang, Z., Pinto-Alva, L., Dernoncourt, F., & Ordóñez, V. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.736722 · Cited by 3
“Anderson et al. (2018) proposed to use off-the-shelf image features from an object detector, and several later works including Huang et al. (2019) and Cornia et al. (2020) follow such design.”
BackgroundA local representation-enhanced recurrent convolutional network for image captioning
Wang, X., & Huang, J. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-022-00231-y · Cited by 3
“Attention mechanisms in image captioning capture linguistic-related visual information from fine-grained visual features [11,22].”
BackgroundMethodMixed Knowledge Relation Transformer for Image Captioning
Chen, T., Li, Z., Wei, J., & Xian, T. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747541 · Cited by 3
MHASAN: Multi-Head Angular Self Attention Network for Spoof Detection
Hasan, M., Roy, K., Rupty, L. K., Hossain, M. S., Sengupta, S., Taus, S. N., & Mohammed, N. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956117 · Cited by 3
“These methods were initially popular in model sequence to sequence problems [26] but are now also used for Visual Computing tasks [27], [28].”
MethodTriple-level relationship enhanced transformer for image captioning
Zheng, A., Zheng, S., Bai, C., & Chen, D. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01073-2 · Cited by 4
“[29] proposed an extended form of attention to complete captioning by constructing a Transformer-like encoder combining with the decoder of the LSTM.”
MethodNested Attention Network with Graph Filtering for Visual Question and Answering
Lu, J. J., Wu, C., Wang, L., Yuan, S., & Wu, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096849 · Cited by 3
“Notice that some recent works apply Attention-on-Attention to image captioning[8] and achieve improvement to fuse different features, which is owing to that this module extends the conventional attention mechanism to determine the correlation between attention results and query results.”
BackgroundAttending to Transforms: A Survey on Transformer-based Image Captioning
Ambilduke, K., Jayakumar, T., Farooqui, L., Padole, H. P., & Singh, A. (2023) · International Conference on Paradigm Shifts in Communications Embedded Systems, Machine Learning and Signal Processing (PCEMS) · doi:10.1109/pcems58491.2023.10136098 · Cited by 3
“AoA [11] extends the conventional self-attention in trans-formers with another attention that is added by an attention gate.”
BackgroundLyu, Y., Liu, Y., & Zhao, Q. (2023) · Electronics · doi:10.3390/electronics12143187 · Cited by 3
A Novel Cross-Fusion Method of Different Types of Features for Image Captioning
Lou, L., Lü, K., & Xue, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191971 · Cited by 3
Dual-feature collaborative relation-attention networks for visual question answering
Lu, Y., Yang, Y., & Hu, J. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00283-8 · Cited by 3
“Rahman et al. [28] introduced the attention on attention [12] module to obtain the relations between questions and images more accurately.”
BackgroundVisual Semantic Relatedness Dataset for Image Captioning
Sabir, A., Moreno-Noguer, F., & Padró, L. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00592 · Cited by 4
“, 2017) has become the new standard for image caption generation systems (Huang et al., 2019; Cornia et al., 2020; Zhang et al., 2021b; Li et al., 2022).”
BackgroundA novel deep fuzzy neural network semantic-enhanced method for automatic image captioning
Vo, T. (2023) · Soft Computing · doi:10.1007/s00500-023-09100-0 · Cited by 3
“Recently, keep going with the utilization of attention mechanism in image captioning, Huang, L. et al. proposed a novel dual attention mechanism called as: AoANet (2019) (Huang et al. 2019).”
BackgroundMethodResultInfluentialA Comprehensive Analysis of Real-World Image Captioning and Scene Identification
(2023) · Journal of Electrical Electronics Engineering · doi:10.33140/jeee.02.03.14 · Cited by 3
Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gender
Sabir, A., & Padró, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.279 · Cited by 5
Research on Image Captioning Based on Vision-language Pre-trained Models
Zou, S., Xie, Y., Yan, J., Wei, Y., & Luan, X. (2023) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia60676.2023.10429361 · Cited by 3
“Model [7] proposed a new attention module called AoA and combined it with the image captioning encoder-decoder architecture, expanding the conventional attention mechanism to better aid in image feature extraction and text generation.”
BackgroundAuxiliary feature extractor and dual attention-based image captioning
Zhao, Q., & Wu, G. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03027-1 · Cited by 3
“Huang et al. [19] presented an AoANet that can more accurately compute the correlation between the vectors and the attention queries, avoiding the problem Fig.”
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
