The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Cited by
Sorted by citations
Items 301–400 of 1,002
Surgical Video Captioning with Mutual-Modal Concept Alignment
Chen, Z., Guo, Q., Yeung, L. K. T., Chan, D. T. M., Lei, Z., Liu, H., & Wang, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43996-4_3 · Cited by 20
UWB-enabled Sensing for Fast and Effortless Blood Pressure Monitoring
Wang, Z., Jin, B., Zhang, F., Li, S., & Ma, J. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659617 · Cited by 20
“The various attention mechanisms have obtained success in computer vision [15, 24, 64], natural language processing [18, 23], also including vital sign monitoring [29, 38].”
BackgroundVisual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
Ge, Y., Zeng, X., Huffman, J. S., Lin, T., Liu, M.-Y., & Cui, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01331 · Cited by 27
“Pioneering works [2, 10, 14] primarily focus on integrating deep neural networks for enhanced image understanding and language generation.”
Zhang, K., Li, P., & Wang, J. (2024) · Remote Sensing · doi:10.3390/rs16214113 · Cited by 30
HERGen: Elevating Radiology Report Generation with Longitudinal Data
Wang, F., Du, S., & Yu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73001-6_11 · Cited by 33
RRGMambaFormer: A hybrid Transformer-Mamba architecture for radiology report generation
Li, H., Liu, S., Wang, H., Jiang, X., Jiu, M., Chen, L., Lu, Y., Lu, Y., & Xu, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127419 · Cited by 19
Caption Generation From Road Images for Traffic Scene Modeling
Li, Y., Wu, C., Li, L., Liu, Y., & Zhu, J. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3072970 · Cited by 19
Latent Memory-augmented Graph Transformer for Visual Storytelling
Qi, M., Qin, J., Huang, D., Shen, Z., Yang, Y., & Luo, J. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475236 · Cited by 19
“As such, most existing approaches for (single) image cap-tioning [9, 14, 16, 22, 25, 32, 53, 56, 66, 75] cannot be directly adopted to handle this task since they neglect the sequential dependencies over the image stream when composing a story with multiple isolated sentences rather than a holistic…”
BackgroundImage Captioning With Positional and Geometrical Semantics
Haque, A. U., Ghani, S., & Saeed, M. (2021) · IEEE Access · doi:10.1109/access.2021.3131343 · Cited by 19
Image Captioning Model Using Part-of-Speech Guidance Module for Description With Diverse Vocabulary
Bae, J.-W., Lee, S.-H., Kim, W.-Y., Seong, J.-H., & Seo, D.-H. (2022) · IEEE Access · doi:10.1109/access.2022.3169781 · Cited by 21
Efficient Image Captioning Based on Vision Transformer Models
Elbedwehy, S., Medhat, T., Hamza, T., & Alrahmawy, M. F. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.029313 · Cited by 19
Sentimental Visual Captioning using Multimodal Transformer
Wu, X., & Li, T. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01752-7 · Cited by 19
“For image captioning, Huang et al. (2019) model the relevance between attention results and queries through an attention on attention module.”
BackgroundMethodImage captioning based on scene graphs: A survey
Jia, J., Ding, X., Pang, S., Gao, X., Xin, X., Hu, R., & Nie, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120698 · Cited by 20
Multimodal Data Augmentation for Image Captioning using Diffusion Models
Xiao, C., Xu, S. X., & Zhang, K. (2023) · doi:10.1145/3607827.3616839 · Cited by 28
“Attention mechanisms [3, 26, 38, 58] and Transformers [9, 34] are actively used and significantly boosting performance.”
BackgroundUnsupervised disease tags for automatic radiology report generation
Yi, X., Fu, Y., Hua, R., Liu, R., & Zhang, H. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105742 · Cited by 19
Image Captioning via Dynamic Path Customization
Ma, Y., Ji, J., Sun, X., Zhou, Y., Hong, X., Wu, Y., & Ji, R. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3409354 · Cited by 20
“…of deep learning, a great improvement can be observed with a flurry of methods [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], e.g., self-critical sequence training (SCST) [43], Up-Down [44], AoANet [45], M 2 Transformer [46], X-LAN [35], and OSCAR [47].”
Context-Driven Image Caption With Global Semantic Relations of the Named Entities
Yun, J., Xu, Z., & Guanglai, G. (2020) · IEEE Access · doi:10.1109/access.2020.3013321 · Cited by 21
Reasoning like Humans: On Dynamic Attention Prior in Image Captioning
Wang, Y., Sun, X., Li, X., Zhang, W., & Gao, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107313 · Cited by 19
Dynamic-balanced double-attention fusion for image captioning
Wang, C., & Gu, X. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105194 · Cited by 19
Yan, T., Shi, J., Li, H., Luo, Z., & Wang, Z. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108629 · Cited by 18
ER-SAN: Enhanced-Adaptive Relation Self-Attention Network for Image Captioning
Li, J., Mao, Z., Fang, S., & Li, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/151 · Cited by 18
“Recently, some researchers focus on variants or modifications of the self-attention operator via using geometric information to enhance region-level features [Huang et al. , 2019; Pan et al. , 2020].”
BackgroundSelf-Learning for Few-Shot Remote Sensing Image Captioning
Zhou, H., Du, X., Xia, L., & Li, S. (2022) · Remote Sensing · doi:10.3390/rs14184606 · Cited by 18
ArCo: Attention-reinforced transformer with contrastive learning for image captioning
Wang, Z., Shi, S., Zhai, Z., Wu, Y., & Yang, R. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104570 · Cited by 18
Learning Double-Level Relationship Networks for image captioning
Wang, C., & Gu, X. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103288 · Cited by 18
Online Public Transit Ridership Monitoring Through Passive WiFi Sensing
Chang, W., Huang, B., Jia, B., Li, W., & Xu, G. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3257835 · Cited by 18
“…time windows have different levels of importances for predicting current ridership, the attention module which can discover associations among different features [37], [38] is integrated into the ConvBiLSTM model to distinguish different impor-tances by automatically assigning different weights.”
MethodImage Captioning with X-Gelu Activated Xgl Transformer
Sharma, D., Dhiman, C., & Kumar, D. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15291-3 · Cited by 19
“Intuitively AoANet [17] enhanced conventional visual attention by providing relevance between the query and the attention results.”
Uncertainty-Aware Image Captioning
Fei, Z., Fan, M., Zhu, L., Huang, J., Wei, X., & Wei, X. K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25137 · Cited by 18
Self-citationJamil, A., Rehman, S. U., Mahmood, K., Villar, M. G., Prola, T. A., Díez, I. D. L. T., Samad, M. A., & Ashraf, I. (2024) · IEEE Access · doi:10.1109/access.2024.3365528 · Cited by 23
“Some image captioning models have employed the original transformer decoder [92]–[95].”
MethodRetractedSelf-Enhanced Attention for Image Captioning
Sun, Q., Zhang, J., Fang, Z., & Gao, Y. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11527-x · Cited by 18
Align and Retrieve: Composition and Decomposition Learning in Image Retrieval With Text Feedback
Xu, Y., Bin, Y., Wei, J., Yang, Y., Wang, G., & Shen, H. T. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3417694 · Cited by 20
“Besides NLP and CV fields, attention has also been heavily used for multimodal applications [20], [21], [56], [57], [58], [59], [60], such as video question answering [20] and image caption [21], [56], [57].”
MethodRegional Attention Network (RAN) for Head Pose and Fine-Grained Gesture Recognition
Behera, A., Wharton, Z., Liu, Y., Ghahremani, M., Kumar, S., & Bessis, N. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3031841 · Cited by 18
“ing attentional models [11], [12], [13] are mostly focused on images with distinctive object categories and/or classes.”
BackgroundInfluentialVisual classification of apple bud-types via attention-guided data enrichment network
Xia, X., Chai, X., Zhang, N., & Sun, T. (2021) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2021.106504 · Cited by 20
Ji, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07061 · Cited by 18
“…(Vaswani et al. 2017), where the self-attention is efficiently utilized to capture the correlations among the regions and words (Liu et al. 2019; Huang et al. 2019; Li et al. 2019a; Herdade et al. 2019; Cornia et al. 2020). is region-biased, which only focuses on the region-level information…”
BackgroundMethodInfluentialFu, Z., Zheng, C., Feng, J., Cai, Y., Wei, X.-Y., Wang, Y., & Li, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3231437 · Cited by 17
“Huang et al. [51] apply an attention-on-attention network to obtain useful knowledge for image captioning.”
BackgroundSelf-citationGenerative image captioning in Urdu using deep learning
Afzal, M. K., Shardlow, M., Tuarob, S., Zaman, F., Sarwar, R., Ali, M., Aljohani, N. R., Lytras, M. D., Nawaz, R., & Hassan, S. (2023) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-023-04584-y · Cited by 21
“Vinyals et al. (2015) and Huang et al. (2019) proposed an “attention on attention” (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and current context.”
BackgroundImproving Image Captioning via Predicting Structured Concepts
Wang, T., Chen, W., Tian, Y., Song, Y., & Mao, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.25 · Cited by 17
Cascade Semantic Prompt Alignment Network for Image Captioning
Li, J., Zhang, L., Zhang, K., Hu, B., Xie, H., & Mao, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3343520 · Cited by 18
“Most of the captioning models [3], [4], [5], [6], [7], [8], [9], [10], [23], [25], [26], [27], [28], [29] use a pre-trained detector encode more fine-grained visual features based Trans-former models.”
BackgroundMethodPolos: Multimodal Metric Learning from Human Feedback for Image Captioning
Wada, Y., Kaneda, K., Saito, D., & Sugiura, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01287 · Cited by 22
“We note that the CapEval1K dataset has several limitations: it is a closed dataset, uses outdated models such as [11, 28, 54], and includes only 1K human judgments.”
MethodCapWAP: Image Captioning with a Purpose
Fisch, A., Lee, K., Chang, M.-W., Clark, J. H., & Barzilay, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.705 · Cited by 31
“Huang et al. (2019) directly optimizes the CIDEr metric with policy gradients, while Luo et al.”
MethodAPSE: Attention-Aware Polarity-Sensitive Embedding for Emotion-Based Image Retrieval
Yao, X., Zhao, S., Lai, Y., She, D., Liang, J., & Yang, J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3042664 · Cited by 17
“The effectiveness of attention mechanism has been demonstrated in various visual tasks, including image captioning [49], [50], person re-identification [51], object detection [52], etc .”
BackgroundAutomatic medical report generation based on deep learning: A state of the art survey
Liu, X., Xin, J., Shen, Q., Huang, Z., & Wang, Z. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102486 · Cited by 22
Rethinking Surgical Captioning: End-to-End Window-Based MLP Transformer Using Patches
Xu, M., Islam, M., & Ren, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_36 · Cited by 17
Dual-View 3D Reconstruction via Learning Correspondence and Dependency of Point Cloud Regions
Jia, X., Yang, S., Wang, Y., Zhang, J., Peng, Y., & Chen, S. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3215024 · Cited by 16
Bottom-up and Top-down Object Inference Networks for Image Captioning
Pan, Y., Li, Y., Yao, T., & Mei, T. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3580366 · Cited by 16
“Inspired by Transformer [51], which achieves impressive performances on a series of NLP tasks, modern image captioning techniques [9, 17] have started to pay more attention to the Transformertype encoder-decoder structure.”
BackgroundMethodDual-Spatial Normalized Transformer for image captioning
Hu, J., Yang, Y., An, Y., & Lu, Y. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106384 · Cited by 16
Context-aware transformer for image captioning
Yang, X. S., Wang, Y., Chen, H., Li, J., & Huang, T. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126440 · Cited by 20
Parallel encoder–decoder framework for image captioning
Saeidimesineh, R., Adibi, P., Karshenas, H., & Darvishy, A. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111056 · Cited by 20
External knowledge-assisted Transformer for image captioning
Li, Z., Su, Q., & Chen, T. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104864 · Cited by 16
SGT++: Improved Scene Graph-Guided Transformer for Surgical Report Generation
Lin, C., Zhu, Z., Zhao, Y., Zhang, Y., He, K., & Zhao, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3335909 · Cited by 19
“And the representative methods based on LSTM are also involved such as SCST [38], AoA [39], X-LAN [40].”
MethodETransCap: efficient transformer for image captioning
Mundu, A., Singh, S. K., & Dubey, S. (2024) · Applied intelligence (Boston) · doi:10.1007/s10489-024-05739-w · Cited by 14
“Hence, existing works in image captioning can be broadly classified into two categories: 1) RNN-Based [11–15], and 2) Transformer-Based [16–23].”
Object Relation Attention for Image Paragraph Captioning
Yang, L.-C., Yang, C.-Y., & Hsu, J. Y.-J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16423 · Cited by 15
“In order to locate highlight objects, several methods are proposed to use attention mechanisms to extract latent region features for text generation and generate improved caption sentences (Xu et al. 2015; Lu et al. 2017; Anderson et al. 2018; Huang et al. 2019).”
MethodNesting spatiotemporal attention networks for action recognition
Li, J., Wei, P., & Zheng, N. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.06.088 · Cited by 16
An Image captioning algorithm based on the Hybrid Deep Learning Technique (CNN+GRU)
Ahmad, R. A., Azhar, M. H., & Sattar, H. (2022) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit57066.2022.00032 · Cited by 16
“[13-16], the encoder (CNN) learns the visual features by”
BackgroundFrom multi-scale grids to dynamic regions: Dual-relation enhanced transformer for image captioning
Zhou, W., Song, C., Chen, D., Su, T., Hu, H., & Shan, C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113127 · Cited by 17
GAVA: Spatial awareness in image captioning with geometric-aware visual attention
Hossain, M. A., Ye, Z., Hossen, M. B., Rahman, M. A., Islam, M. S., & Abdullah, M. I. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105435 · Cited by 16
Hierarchical Region-Context Attention for image captioning
Hossain, M. A., Ye, Z., Hossen, M. B., Rahman, M. M., Islam, M. S., & Abdullah, M. I. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114014 · Cited by 16
Image Captioning with Attention for Smart Local Tourism using EfficientNet
Fudholi, D. H., Windiatmoko, Y., Afrianto, N., Susanto, P. E., Suyuti, M., Hidayatullah, A. F., & Rahmadi, R. (2021) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/1077/1/012038 · Cited by 15
#PraCegoVer: A Large Dataset for Image Captioning in Portuguese
dos Santos, G. O., Colombini, E. L., & Avila, S. (2022) · Data · doi:10.3390/data7020013 · Cited by 18
“To validate our dataset and have a benchmark, we carried out experiments with AoANet [12], one of the state-of-art algorithms for MS COCO Captions.”
MethodResultInfluentialKnowledge-Enriched Attention Network With Group-Wise Semantic for Visual Storytelling
Li, T., Wang, H., He, B., & Chen, C. W. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3230934 · Cited by 20
“features, where convolutional features [1], [2], [3] and object”
BackgroundMulti-modal adaptive gated mechanism for visual question answering
Xu, Y., Zhang, L., & Shen, X. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287557 · Cited by 18
Crossing the Gap: Domain Generalization for Image Captioning
Ren, Y., Mao, Z., Fang, S., Lu, Y., He, T., Du, H., Zhang, Y., & Ouyang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00281 · Cited by 16
“We evaluate several most popular SOTA image captioning methods, including Up-Down [3], AoANet [26], M(2)Transformer [13].”
BackgroundMethodInfluentialZeng, Q., Sun, J., & Wang, S. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2023.1273029 · Cited by 15
CA-Captioner: A novel concentrated attention for image captioning
Yang, X., Yang, Y., Wu, J., Sun, W., Ma, S., & Hou, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123847 · Cited by 16
Multi-Modal Graph Aggregation Transformer for image captioning
Chen, L., & Li, K. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106813 · Cited by 16
Hossain, M. S., Aktar, S., Gu, N., Liu, W., & Huang, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126692 · Cited by 15
Zhou, W., Jiang, W., Zheng, Z., Li, J., Su, T., & Hu, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126850 · Cited by 15
Image captioning with semantic-enhanced features and extremely hard negative examples
Cai, W., & Liu, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.112 · Cited by 15
Bidirectional Mapping Coupled GAN for Generalized Zero-Shot Learning
Shermin, T., Teng, S. W., Sohel, F. A., Murshed, M., & Lu, G. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3135480 · Cited by 14
“noise and non-discriminative information in the embedding [3], [8], [27], [51], [53], a few works have applied attention mechanisms [24], [33], [34], [36], [37], [54]–[57].”
BackgroundEnhancing Descriptive Image Captioning with Natural Language Inference
Shi, Z., Liu, H., & Zhu, X. (2021) · doi:10.18653/v1/2021.acl-short.36 · Cited by 15
“AoANet (Huang et al., 2019) adopts the attention on attention module.”
MethodAGNet: Automatic generation network for skin imaging reports
Wu, F., Yang, H., Peng, L., Lian, Z., Li, M., Qu, G., Jiang, S., & Yu, H. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105037 · Cited by 16
TAP: Text-Aware Pre-training for Text-VQA and Text-CaptionPreprint
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04638 · Cited by 15
“8 AoANet [22] 42.”
Automated Generation of Accurate \& Fluent Medical X-ray ReportsPreprint
Nguyen, H. T., Nie, D., Badamdorj, T., Liu, Y., Zhu, Y., Truong, J., & Li, C. (2021) · arXiv (Cornell University) · Cited by 15
Video captioning: A comparative review of where we are and which could be the route
Moctezuma, D., Ramírez-delReal, T. A., Ruíz, G., & González-Chávez, O. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103671 · Cited by 18
Transformer model incorporating local graph semantic attention for image caption
Qian, K., Pan, Y., Xu, H., & Tian, L. (2023) · The Visual Computer · doi:10.1007/s00371-023-03180-7 · Cited by 14
“Huang [35] designed an Attention on Attention (AoA) module, which extended the conventional attention mechanisms to determine the relevance between attention results and queries.”
Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
Özdemir, Ö., & Akagündüz, E. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00163 · Cited by 19
“Conventional approaches in image captioning are based on attention and encoder-decoder structure [13, 14, 41].”
BackgroundCM-SC: Cross-modal spatial-channel attention network for image captioning
Hossain, M. S., Aktar, S., Hossain, M. A., Gu, N., & Huang, Z. (2024) · Displays · doi:10.1016/j.displa.2024.102941 · Cited by 15
Linguistically-aware attention for reducing the semantic gap in vision-language tasks
Gouthaman, K. V., Nambiar, A., Srinivas, K. S., & Mittal, A. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107812 · Cited by 15
“A recent work [12] proposed a complex at-”
Revisiting image captioning via maximum discrepancy competition
Wan, B., Jiang, W., Fang, Y., Zhu, M., Li, Q., & Liu, Y. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108358 · Cited by 14
Explicit Image Caption Editing
Wang, Z., Chen, L., Ma, W., Han, G., Niu, Y., Shao, J., & Xiao, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_7 · Cited by 14
Facilitated Deep Learning Models for Image Captioning
Azhar, I., Afyouni, I., & Elnagar, A. (2021) · Conference on Information Sciences and Systems · doi:10.1109/ciss50987.2021.9400209 · Cited by 14
“Other studies on the effects of attention in image captioning have promoted its usage quite heavily [8]–[10], by measuring the ground truth for attention map and the labeling of image regions for both Flickr30k and MS COCO datasets.”
BackgroundExploring Explicit And Implicit Visual Relationships For Image Captioning
Song, Z., & Zhou, X. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428310 · Cited by 14
“On the other hand, many studies [9, 12, 13] employ self-attention based transformer to adaptively learn relevent weights between each object pairs.”
BackgroundMethodInfluentialThe Curious Layperson: Fine-Grained Image Recognition Without Expert Labels
Choudhury, S., Laina, I., Rupprecht, C., & Vedaldi, A. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01885-9 · Cited by 13
ArtCap: A Dataset for Image Captioning of Fine Art Paintings
Lu, Y., Guo, C., Dai, X., & Wang, F. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3223539 · Cited by 14
“Considering that the employing of a Transformer is an important and effective method in image captioning in recent years, we employ our dataset to train and test a Transformer-based image cap-tioning model (AoANet) [30].”
BackgroundMethodInfluentialVeg-DenseCap: Dense Captioning Model for Vegetable Leaf Disease Images
Sun, W., Wang, C., Gu, J., Sun, X., Li, J., & Liang, F. (2023) · Agronomy · doi:10.3390/agronomy13071700 · Cited by 13
“[19] proposed an Attention-to-Attention (AoA) module, which is applied to both the encoder and decoder to obtain the relationship between different objects in the image and filter out irrelevant content, so as to improve the smoothness of image captioning.”
MethodHierarchical Semantic Enhancement Network for Multimodal Fake News Detection
Zhang, Q., Liu, J., Zhang, F., Xie, J., & Zha, Z.-J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612423 · Cited by 15
Tang, Y., Wang, D., Zhang, L., & Yuan, Y. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105651 · Cited by 14
Gu, Y., Fang, L., Mou, L., Ma, S., Yan, Q., Zhang, J., Liu, F., Liu, J., & Zhao, Y. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105836 · Cited by 13
Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning
Hafeth, D. A., & Kollias, S. (2024) · Sensors · doi:10.3390/s24061796 · Cited by 18
“Huang et al. [12] proposed AoANet, introducing an additional attention mechanism by employing gating on the information, thereby enhancing the model’s ability to focus on relevant information.”
BackgroundExploring refined dual visual features cross-combination for image captioning
Hu, J., Li, Z., Su, Q., Tang, Z., & Ma, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106710 · Cited by 13
DeepLabv3Att: Integrating Attention Mechanisms in DeepLabv3 for Enhanced Road Segmentation
Hossain, M. S., Rahman, M., Ahmed, M., Kabir, M. M., Mridha, M. F., & Shin, J. (2024) · International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT) · doi:10.1109/3ict64318.2024.10824285 · Cited by 13
“The attention mechanism allows models to focus on relevant parts of the input data, leading to improved performance in tasks such as machine translation [20], image captioning [21], and image classification [22].”
A Survey on Enhancing Image Captioning with Advanced Strategies and Techniques
Zou, B., Thobhani, A., Kui, X., Abdussalam, A., Asim, M., Shah, S., & ELAffendi, M. A. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.059192 · Cited by 15
Image-Captioning Model Compression
Atliha, V., & Šešok, D. (2022) · Applied Sciences · doi:10.3390/app12031638 · Cited by 13
Conditional Embedding Pre-Training Language Model for Image Captioning
Li, P., Zhang, M., Lin, P., Wan, J., & Jiang, M. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10844-3 · Cited by 13
“Recently, studies using transformer structures have also gradually increased, such as AoANet [51], NG-SAN [26], and ORT [55].”
BackgroundInfluentialLayer-wise enhanced transformer with multi-modal fusion for image caption
Li, J., Wang, Y., & Zhao, D. (2022) · Multimedia Systems · doi:10.1007/s00530-022-01036-z · Cited by 14
“AOA [13] extends the conventional attention mechanism to determine the correlation between attention results and query results, and finally obtain the expected useful knowledge.”
BackgroundMethodInfluentialAlsayed, A., Arif, M., Qadah, T. M., & Alotaibi, S. S. (2023) · Applied Sciences · doi:10.3390/app131910894 · Cited by 15
TSFNet: Triple-Steam Image Captioning
Hu, N., Ming, Y., Fan, C., Feng, F., & Lyu, B. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3215861 · Cited by 13
Attention in Reasoning: Dataset, Analysis, and Modeling
Chen, S., Jiang, M. X., Yang, J., & Zhao, Q. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3114582 · Cited by 13
“Attention, as an information selection mechanism, has been widely used in various DNN models [1], [2], [3], [4], [5], [6], [6], [7], [8], [9], to improve their ability of localizing important parts of the inputs, as well as task performance.”
MethodPerformance Comparison of AI Models for Digital Image captioning
Kaur, R., & Singh, G. (2022) · International Conference on Computational Modelling, Simulation and Optimization (ICCMSO) · doi:10.1109/iccmso58359.2022.00058 · Cited by 12
“al [9]proposed a module named attention on attention and was basically an addition to standard attention mechanisms to discover relation between attention results and queries.”
BackgroundComplete 3D Relationships Extraction Modality Alignment Network for 3D Dense Captioning
Mao, A., Yang, Z. X., Chen, W., Yi, R., & Liu, Y. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3279204 · Cited by 12
“Huang et al. [21] augments the traditional self-attention mechanism by addressing irrelevant attention concerns.”
BackgroundAre metrics measuring what they should? An evaluation of Image Captioning task metrics
González-Chávez, O., Ruíz, G., Moctezuma, D., & Ramírez-delReal, T. A. (2023) · Signal Processing Image Communication · doi:10.1016/j.image.2023.117071 · Cited by 15
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
