The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Attention on Attention for Image Captioning
Open accessAbstract
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DropToken [174] reduces the training complexity via random dropping a portion of the video and audio tokens from input sequence during training.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [174]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Cited by
Sorted by citations
Items 101–200 of 1,002
Dual Attention on Pyramid Feature Maps for Image Captioning
Yu, L., Zhang, J., & Wu, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3072479 · Cited by 63
“Different from the attention model [Huang et al., 2019a] mainly focuses on language generation, we try to explore visual feature representations for image captioning.”
BackgroundMethodInfluentialDual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning
Dong, X., Long, C., Xu, W., & Xiao, C. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475439 · Cited by 74
“Compared with the latest AOA method [21], our model produces better BLEU-1 and CIDEr scores, which are 1.”
BackgroundMethodNeural attention for image captioning: review of outstanding methods
Zohourianshahzadi, Z., & Kalita, J. K. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10092-2 · Cited by 68
“Types of encoder could be (CNN) or object detector with CNN (OD), or a combination of these with (GCN) or Attention-on-Attention (AoA) [40].”
InfluentialGLCM: Global–Local Captioning Model for Remote Sensing Image Captioning
Wang, Q., Huang, W. F., Zhang, X., & Li, X. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3222606 · Cited by 62
“Inspired by natural image captioning (NIC) [28], [29], [30], [31], [32], [33], [34], [35], Qu et al. [14] first introduced CNN + RNN/LSTM mechanism in RSIC, publishing two datasets of UCM-captions and Sydney-captions.”
MethodTransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing
Xu, Y., Yin, Y., Jiang, L., Wu, Q., Zheng, C., Loy, C. C., Dai, B., & Wu, W. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00753 · Cited by 63
“Multi-head attention module is often used in some multimodality tasks to establish the inter-modal interaction between text modalities and visual modalities, such as image captioning [7, 17] and text-to-image translation [8, 30].”
BackgroundAttentive Contextual Network for Image Captioning
Prudviraj, J., Vishnu, C., & Mohan, C. K. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533970 · Cited by 64
Vision-Enhanced and Consensus-Aware Transformer for Image Captioning
Cao, S., An, G., Zheng, Z., & Wang, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3178844 · Cited by 63
TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation
Yi, X., Fu, Y., Liu, R., Zhang, H., & Hua, R. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3350077 · Cited by 64
A survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59
Retrieval-Augmented Transformer for Image Captioning
Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2022) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1145/3549555.3549585 · Cited by 59
“Following standard image captioning approaches [3, 16, 27], we train and evaluate our model on the COCO dataset [38], thus not relying on large-scale image-text datasets [15].”
BackgroundMethodLabel-attention transformer with geometrically coherent objects for image captioning
Dubey, S., Olimov, F., Rafique, M. A., Kim, J., & Jeon, M. (2022) · Information Sciences · doi:10.1016/j.ins.2022.12.018 · Cited by 59
“LATGeO outperforms DNN, attention-based models, and transformer-based algorithms [3], [4], [39], [40] in most of the evaluation metrics.”
BackgroundMethodMeaCap: Memory-Augmented Zero-shot Image Captioning
Zeng, Z., Xie, Y., Zhang, H., Chen, C., Chen, B., & Wang, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01337 · Cited by 75
“For better visual understanding, some methods [3, 7, 17, 18, 26, 43, 59] employ an object detector to extract attentive image regions.”
Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning
Wang, J., Tang, J., & Luo, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413753 · Cited by 57
“When digging into the generated sentences from Up-Down and AoA, it is found that even though the models can describe the objects that appeared in images well, they cannot include image text in descriptions.”
BackgroundInfluentialImage Change Captioning by Learning from an Auxiliary Task
Hosseinzadeh, M., & Wang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00275 · Cited by 62
“Visual attention has been proved to be an effective technique for the image captioning models and has been used extensively since then [5, 4, 12, 24].”
BackgroundMethodAdaptive Semantic-Enhanced Transformer for Image Captioning
Zhang, J., Fang, Z., Sun, H., & Wang, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3185320 · Cited by 59
“Huang et al. [43] propose an attention on attention network (AoANet) that adds another attention by applying the gate to the multihead attention.”
MethodInfluentialMemory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation
Divya, P., Yenduri, S., Vishnu, C., Mohan, C. K., & Chen, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3371894 · Cited by 62
“Further, in [40], a stack of self-attention modules captures intricate visual feature relationships, and an attention gate filters out irrelevant information, enhancing caption quality.”
BackgroundSAMT-generator: A second-attention for image captioning based on multi-stage transformer network
Yang, X., Yang, Y., Ma, S., Li, Z., Dong, W. F., & Woźniak, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127823 · Cited by 55
Attention-Based Fully Gated CNN-BGRU for Russian Handwritten Text
Abdallah, A., Hamada, M., & Nurseitov, D. B. (2020) · Journal of Imaging · doi:10.3390/jimaging6120141 · Cited by 57
“Inspired by the latest advances in machine translation [17], automated question answer [18], image captioning [19], sentimental text processing[20],and speech recognition [21], we believe that encoder-decoder models with attention mechanisms [22, 23] will become the new state-of-the-art for HTR tasks.”
Context-Aware Graph Inference With Knowledge Distillation for Visual Dialog
Guo, D., Wang, H., & Wang, M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3085755 · Cited by 53
“…between vision and language has attracted more and more interests, such as referring expression [1], [2], [3], [4], [5], visual captioning [6], [7], [8], [9], [10], [11], visual question answering (VQA) [12], [13], [14], [15], [16], [17], and image/video-text retrieval [18], [19]…”
BackgroundMultiscale Multiinteraction Network for Remote Sensing Image Captioning
Wang, Y., Zhang, W., Zhang, Z., Gao, X., & Sun, X. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3153636 · Cited by 56
ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing
Zeng, Z., Zhang, H., Lu, R., Wang, D., Chen, B., & Wang, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02247 · Cited by 56
“To encourage more interactions between two modalities, attention mechanism [17, 35, 50, 51, 59, 60] and graph neural network [76,77] have been widely adopted.”
MethodEnd-to-End 3D Dense Captioning with Vote2Cap-DETR
Chen, S., Zhu, H., Chen, X., Lei, Y., Yu, G., & Chen, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01070 · Cited by 62
Generating Radiology Report Via Auxiliary Signal Guidance and Memory-Driven Network
Xue, Y., Tan, Y., Tan, L., Qin, J., & Xiang, X. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121260 · Cited by 55
Prompt-Based Learning for Unpaired Image Captioning
Zhu, P., Wang, X., Zhu, L., Sun, Z., Zheng, W., Wang, Y., & Chen, C. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3265842 · Cited by 57
“…different strategies, including attention mechanisms (salient objects attention [35], bottom-up and top-down attention [36], attention on attention [37], X-linear attention [38], fully-attentive model [39]), focus of scene semantics [40], [41] (objects [42], relations [43], and attributes [44]),…”
BackgroundSharma, H., & Padha, D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10488-2 · Cited by 51
“Huang et al. (2021) devised a de-noising approach for visual feature extraction to address this issue. According to Li et al. (2021), the crossentropy objective function employed by earlier RSIC frameworks attempts to train the model to predict the following word with exact probability only. However, there may be a va…”
BackgroundMethodInfluentialUnpaired Image Captioning With semantic-Constrained Self-Learning
Ben, H., Pan, Y., Li, Y., Yao, T., Hong, R., Wang, M., & Mei, T. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3060948 · Cited by 54
Park, H., Kim, K. M., Park, S., & Choi, J. (2021) · IEEE Access · doi:10.1109/access.2021.3124564 · Cited by 52
Knowing What to Learn: A Metric-Oriented Focal Mechanism for Image Captioning
Ji, J., Ma, Y., Sun, X., Zhou, Y., Wu, Y., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3183434 · Cited by 50
“Existing IC models treat all training examples equally, and barely consider the imbalance between easy and hard examples [5], [9], [10], [35], [36].”
BackgroundMethodInfluentialInceptr: micro-expression recognition integrating inception-CBAM and vision transformer
Zhou, H., Huang, S., & Xu, Y. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01164-0 · Cited by 53
Yang, Z., Li, Q., Yuan, Y., & Wang, Q. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3401576 · Cited by 59
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Self-citationImage captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49
Li, J., Zhang, Y., Qiang, W., Si, L., Jiao, C., Hu, X., Zheng, C., & Sun, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25216 · Cited by 50
Cross-Modal Hybrid Feature Fusion for Image-Sentence Matching
Xu, X., Wang, Y., He, Y., Yang, Y., Hanjalić, A., & Shen, H. T. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3458281 · Cited by 50
“Image-Sentence Matching [10, 22, 33, 44, 61] is a fundamental task, which combines the visual and linguistic information from crossmodal data of images and sentences, and facilitates other applications such as visual question answering (VQA) [2, 14, 65], image captioning [1, 19, 50], and visual relation detection [3],…”
BackgroundImage Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge
Dognin, P., Melnyk, I., Mroueh, Y., Padhi, I., Rigotti, M., Ross, J., Schiff, Y., Young, R. A., & Belgodere, B. M. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13113 · Cited by 56
Visual Cluster Grounding for Image Captioning
Jiang, W., Zhu, M., Fang, Y., Shi, G., Zhao, X., & Liu, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3177318 · Cited by 50
Improving OCR-based Image Captioning by Incorporating Geometrical Relationship
Wang, J., Tang, J., Yang, M., Bai, X., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00136 · Cited by 49
“Most recently, the explorations are mainly along with the directions of learning the relationship between objects [40], improving the attention mechanism [14, 28] and adapting Transformer as the decoder [18, 10, 25].”
BackgroundSemi-Autoregressive Transformer for Image Captioning
Zhou, Y., Zhang, Y., Hu, Z., & Wang, M. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00350 · Cited by 50
“On the sentence decoding side, we have witnessed LSTM [24], CNN [8] and Transformer [4] equipped with various attention [12, 31, 18] as decoder.”
MethodMMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning
Zheng, S., Wang, W., Qu, J., Yin, H., Chen, W., & Zhao, L. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00015 · Cited by 51
“Here, we set the number of epochs E ∈ [10, 30, 50, 70, 90, 110] and the size of batches N ∈ [16, 32, 64, 128, 256, 512].”
BackgroundSelf-citationRetrieval-augmented Image Captioning
Ramos, R., Elliott, D., & Martins, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.266 · Cited by 61
A Study of ConvNeXt Architectures for Enhanced Image Captioning
Ramos, L., Casas, E., Romero, C., Rivas, F., & Morocho-Cayamcela, M. E. (2024) · IEEE Access · doi:10.1109/access.2024.3356551 · Cited by 47
“To address these shortcomings, attentional mechanisms have been introduced, enabling the model to focus on salient parts of the image during caption generation, thereby producing more contextually accurate and human-like descriptions [7], [13].”
Scaling-up medical vision-and-language representation learning with federated learning
Lu, S., Liu, Z., Liu, T., & Zhou, W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107037 · Cited by 49
Zhang, S., Li, Y., & Mei, S. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3333375 · Cited by 46
Multi-caption Text-to-Face Synthesis: Dataset and Algorithm
Sun, J.-X., Li, Q., Wang, W.-N., Zhao, J., & Sun, Z.-N. (2021) · ACM Multimedia · doi:10.1145/3474085.3475391 · Cited by 64
“In recent years, attention mechanism has been widely exploited in the intersection between vision and language modalities [2, 9, 25, 27, 29].”
BackgroundSelf-citationExploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning
Hu, J. C., Cavicchioli, R., & Capotondi, A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386812 · Cited by 54
“Most modern Image Captioning systems are currently based on the Transformer architecture [10] and many works focused on improving its formulation or structure [14], [17], [19], [31]–[34].”
BackgroundMethodImproving Image Captioning Evaluation by Considering Inter References Variance
Yi, Y., Deng, H., & Hu, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.93 · Cited by 43
“It is worth noting that different judgments exist between AoAnet and M2-Transformer on our metric and CIDEr-D.”
MethodDetection of River Floating Garbage Based on Improved YOLOv5
Yang, X., Zhao, J., Zhao, L., Zhang, H., Li, L., Ji, Z., & Ганчев, И. (2022) · Mathematics · doi:10.3390/math10224366 · Cited by 45
“In the learning process of a neural network, the larger the amount of information stored by the model, the better the expressiveness of the model [28].”
BackgroundPositive-Augmented Contrastive Learning for Image and Video Captioning Evaluation
Sarto, S., Barraco, M., Cornia, M., Baraldi, L., & Cucchiara, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00668 · Cited by 54
“As such, the task has witnessed methodological and architectural innovations, ranging from the usage of self-attentive models [10, 16, 19, 36] to the development of better connections between visual and textual modalities with the addition of objects [3, 62, 66] and tags [29, 63] or the use of more powerful cross-moda…”
BackgroundZhou, Y. Z., Chen, S., Zhao, J., Yao, R., Xue, Y., & Saddik, A. E. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3204770 · Cited by 43
“In image captioning [22], [23], [24], a general detector extracted and saved multiobjects’ features which were input to the transformer after RoIPooling [1].”
MethodCaMEL: Mean Teacher Learning for Image Captioning
Barraco, M., Stefanini, M., Cornia, M., Cascianelli, S., Baraldi, L., & Cucchiara, R. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9955644 · Cited by 45
Controllable Image Captioning via Prompting
Wang, N., Xie, J., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25360 · Cited by 50
Compare and Reweight: Distinctive Image Captioning Using Similar Images Sets
Wang, J., Xu, W., Wang, Q., & Chan, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_22 · Cited by 42
“For vision-language connection, attention mechanisms [41,29,2,16] are used to reveal the co-occurrence between concepts and objects in the images.”
Exploring region relationships implicitly: Image captioning with visual relationship attention
Zhang, Z., Wu, Q., Wang, Y., & Chen, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104146 · Cited by 43
Stacked Multimodal Attention Network for Context-Aware Video Captioning
Zheng, Y., Zhang, Y., Feng, R., Zhang, T., & Fan, W. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3058626 · Cited by 45
Bi-Directional Co-Attention Network for Image Captioning
Jiang, W., Wang, W., & Hu, H. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3460474 · Cited by 42
“For the offline evaluation, the proposed BCAN is compared with the existing state-of-the-art methods including the Adaptive [28], Att2in [37], Updown [2], NBT [29], HAN [42], RFNet [15], GCN-LSTM [47], HIP [48], AoANet [14], and M 2 [6] on the Karpathy’s test set.”
BackgroundMethodSelf-citationExplaining transformer-based image captioning models: An empirical analysis
Cornia, M., Baraldi, L., & Cucchiara, R. (2021) · AI Communications · doi:10.3233/aic-210172 · Cited by 42
“…development of effective algorithms working at the intersection of the Computer Vision and Natural Language Processing domains, with applications such as image and video captioning [2,14,24,27], multi-modal retrieval [13,18,34,51], visual question answering [63,68,85], and embodied AI [3,29,32].”
InfluentialAutomated Generation of Accurate & Fluent Medical X-ray Reports
Nguyen, H. N., Nie, D., Badamdorj, T., Liu, Y., Zhu, Y., Truong, J., & Li, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.288 · Cited by 60
“…captioning, which aims at generating realistic sentences or topic-related paragraphs to summarize visual contents from images or videos (Vinyals et al., 2015; Xu et al., 2015; Goyal et al., 2017; Rennie et al., 2017; Huang et al., 2019; Feng et al., 2019; Pei et al., 2019; Tran et al., 2020).”
BackgroundUnifying Relational Sentence Generation and Retrieval for Medical Image Report Composition
Wang, F., Liang, X., Xu, L. H., & Lin, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3026098 · Cited by 43
“Besides the above methods, we also compare with state-of-the-art methods of medical report composition, including AoA [40], Transformer [35], BUTD [10], CoAtt [2], HRGR-Agent [1], and KERP [29].”
InfluentialUncertainty-aware report generation for chest X-rays by variational topic inference
Najdenkoska, I., Zhen, X., Worring, M., & Shao, L. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102603 · Cited by 43
“For instance, Huang et al. (2019) suggests to extend the conventional attention mechanism to determine the relevance between attention results and queries, denoted as the attended information or the expected useful knowledge.”
DeeCap: Dynamic Early Exiting for Efficient Image Captioning
Fei, Z., Yan, X., Wang, S., & Tian, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01190 · Cited by 41
“To be consistent with previous works [10,22], we convert all the descriptions to lower case and omit words which occur less than 5 times.”
BackgroundMethodHAAV: Hierarchical Aggregation of Augmented Views for Image Captioning
Kuo, C.-W., & Kira, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01062 · Cited by 45
“Other works also propose to use a sigmoid/tanh gating mechanism [22] to aggregate across layers [10], which can be generalized to aggregating across views.”
BackgroundMethodInfluentialLarge Foundation Model Empowered Region-aware Underwater Image Captioning
Li, H., Li, L., Wang, H. E., Zhang, W., & Ren, P. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02650-w · Cited by 41
UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning
Lee, H., Yoon, S., Dernoncourt, F., Bui, T., & Jung, K. (2021) · doi:10.18653/v1/2021.acl-short.29 · Cited by 48
Memorial GAN With Joint Semantic Optimization for Unpaired Image Captioning
Song, P., Guo, D., Zhou, J., Xu, M., & Wang, M. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3175012 · Cited by 41
“Image captioning has made remarkable achievements in recent years [5], [28]–[30].”
BackgroundFashion Image Retrieval with Text Feedback by Additive Attention Compositional Learning
Tian, Y., Newsam, S. D., & Boakye, K. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00107 · Cited by 44
“The concept of attention has gained popularity recently in neural networks as it allows the models to learn representations from different modalities [33, 27, 20, 14, 5, 18].”
BackgroundLin, Z., Zhang, D., Shi, D., Xu, R., Tao, Q., Wu, L., He, M., & Ge, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104281 · Cited by 39
Multi-Branch Distance-Sensitive Self-Attention Network for Image Captioning
Ji, J., Huang, X., Sun, X., Zhou, Y., Luo, G., Cao, L., Liu, J., Shao, L., & Ji, R. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3169061 · Cited by 39
“The baseline models we compare include SCST [32], LSTM-A [58], Up-Down [8], RFNet [55], GCN-LSTM [24], SGAE [25], ETA [35], AoANet [9], ORT [16], MMTransformer [11], XTransformer [10], and RSTNet [56].”
MethodSPT: Spatial Pyramid Transformer for Image Captioning
Zhang, H., Zeng, P., Gao, L., Lyu, X., Song, J., & Shen, H. T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3336371 · Cited by 44
“Recently, various methods [12], [18], [27] have designed several sophisticated attention modules to interact with important visual and text features effectively.”
BackgroundMethodDCMSTRD: End-to-end Dense Captioning via Multi-Scale Transformer Decoding
Shao, Z., Han, J., Debattista, K., & Pang, Y. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3369863 · Cited by 41
“For example, [12] proposed an Attention on Attention model, which extended the self-attention in the Transformer to optimise the results of attention.”
MethodUmirzakova, S., Muksimova, S., Mardieva, S., Sultanov, M., & Cho, Y. I. (2024) · Sensors · doi:10.3390/s24248013 · Cited by 39
Attention in Attention Networks for Person Retrieval
Fang, P., Zhou, J., Roy, S. K., Ji, P., Petersson, L., & Harandi, M. T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3073512 · Cited by 39
“Thereafter, several visual applications have incorporated this attention module in their formulation, e.g., image classification [43], scene segmentation [44], image captioning [45] as well as video person re-ID [41].”
MethodKnowing What it is: Semantic-Enhanced Dual Attention Transformer
Ma, Y., Ji, J., Sun, X., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3164787 · Cited by 39
“We compare the results of ensembled models with different SOTAs, including: SCST [3], Transformer [32], Up-Down [4], RFNet [74], GCN-LSTM [75], GAE [76], SGAE [76], ETA [80], AoANet [77], ORT [78],M2Transformer [6], XTransformer [7] and GCN-LSTM+HIP [81].”
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning
Biten, A. F., Gomez, L., & Karatzas, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00253 · Cited by 43
“We use Adam [22] optimizer with batch size 10 and learning rate 0.0002 and 0.0005 for UpDown [3] and AoA [19], respectively.”
InfluentialHu, N., Fan, C., Ming, Y., & Feng, F. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.11.045 · Cited by 38
Auto-Parsing Network for Image Captioning and Visual Question Answering
Yang, X., Gao, C., Zhang, H., & Cai, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00220 · Cited by 38
“, AoANet [20],M(2) Transformer [16] and X-LAN [44] in the future.”
BackgroundInfluentialGilBERT: Generative Vision-Language Pre-Training for Image-Text Retrieval
Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., & Chu, W. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462838 · Cited by 37
Self-citationUnifying Multimodal Transformer for Bi-directional Image and Text Generation
Huang, Y., Xue, H., Liu, B., & Lu, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3481540 · Cited by 41
Sharma, D., Dhiman, C., & Kumar, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119773 · Cited by 43
Underwater Image Captioning With AquaSketch-Enhanced Cross-Scale Information Fusion
Li, H., Li, L., Wang, H., Zhang, W., & Ren, P. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3585119 · Cited by 39
“These include RNN-based methods such as NIC [12], Soft-Att [79], Bi-LSTM [28], Spatial-Att [80], Adaptive-Att [80], Up-Down [81], and AoANet [82], as well as Transformer-based methods like Transformer [36], M2 [83], DLCT [43], RSTNet [84], S2 [78], LSTNet [85], SDATR [86], MDSANet [87], DTNet [44],…”
Explain and improve: LRP-inference fine-tuning for image captioning models
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.008 · Cited by 52
“We will analyze the adaptive attention mechanism [7], [8], [9], [10] and the multi-head attention mechanism [11], [36], [12], [13], [14], [15].”
BackgroundMethodInfluentialImproving image captioning with Pyramid Attention and SC-GAN
Chen, T., Li, Z., Wu, J., Ma, H., & Su, B. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104340 · Cited by 37
Transforming remote sensing images to textual descriptions
Zia, U., Riaz, M. M., & Ghafoor, A. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102741 · Cited by 42
Attention-Aligned Transformer for Image Captioning
Fei, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19940 · Cited by 37
“All the training sentences are pre-posed by converting them into lower case and dropping the words that occur rarely as (Huang et al. 2019; Cornia et al. 2020).”
BackgroundMethodProgressive Tree-Structured Prototype Network for End-to-End Image Captioning
Zeng, P., Zhu, J., Song, J., & Gao, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548024 · Cited by 37
“The comparison methods can be roughly divided into three categories: i) Two-stage methods, which adopt offline features directly to reason descriptions, including AoANet [15],M2 Transformer [5], GET [17], X-Transofmer [27], DRT [35], RSTNet [52] and DLCT [25]; ii) One-stage methods, which optimize feature extraction m…”
MethodUtilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports
Zhu, Q., Mathai, T. S., Mukherjee, P., Peng, Y., Summers, R. M., & Lu, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_19 · Cited by 38
“The same Longitudinal-MIMIC dataset was used to train all baseline models, such as AoANet [16], CNNTrans [14], Transformer [13], R2gen [7], and R2CMN [8].”
Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation
Goel, A., Fernando, B., Keller, F., & Bilen, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01515 · Cited by 36
“SGG can be seen as a step towards comprehensive scene understanding and benefits several high-level visual tasks such as object detection/segmentation [17, 20, 41], image captioning [1, 18, 22, 30], image/video retrieval [16], and visual question answering [2, 34].”
A CNN-transformer hybrid approach for decoding visual neural activity into text
Zhang, J., Li, C., Liu, G., Min, M., Wang, C., Li, J., Wang, Y., Yan, H., Zuo, Z., Huang, W., et al. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106586 · Cited by 37
Progressive Learning for Image Retrieval with Hybrid-Modality Queries
Zhao, Y., Song, Y., & Jin, Q. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532047 · Cited by 39
“VQA [1, 2, 11] and image captioning [1, 17, 33], both the single-stream and twostream methods suffer from the low retrieval efficiency problem due to the cross fusion encoder, and thus are inappropriate for the image retrieval task.”
MethodNICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Models
Chen, S., Song, Z., Haque, M., Liu, C., & Yang, W. P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01493 · Cited by 38
“Designing NICG models is a challenging task but could have a massive impact in the real world [1, 7, 22, 30], such as helping people with visual impairment to understand visual inputs, enhancing the accuracy of image search engines, or transferring images to text/audio in social media, etc.”
BackgroundA Multiscale Grouping Transformer With CLIP Latents for Remote Sensing Image Captioning
Meng, L., Wang, J., Meng, R., Yang, Y., & Xiao, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3385500 · Cited by 42
“Huang et al. [38] proposed an attention on attention (AoA) that enhances self-attention by examining the correlation between attention outputs and queries.”
BackgroundMethodRecurrent Relational Memory Network for Unsupervised Image Captioning
Guo, D., Wang, Y., Song, P., & Wang, M. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/128 · Cited by 38
“Traditional image captioning [ Yao et al. , 2019; Huang et al. , 2019 ] requires full supervision of image-caption pairs annotated by humans.”
BackgroundBoost image captioning with knowledge reasoning
Huang, F., Li, Z., Wei, H., Zhang, C., & Ma, H. (2020) · Machine Learning · doi:10.1007/s10994-020-05919-y · Cited by 35
“[13] devised an AoA (Attention on Attention) model to filter out the inappropriate attention results, and only used the useful attended information to guide the caption generation process.”
MethodWang, S., Zeng, Q., Ni, W., Cheng, C., & Wang, Y. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.107863 · Cited by 35
Zou, Y., Wu, H., Yin, Y., Dhamotharan, L., Chen, D., & Tiwari, A. K. (2022) · Annals of Operations Research · doi:10.1007/s10479-022-04788-z · Cited by 34
“Tosurroundthis problem,weexploittheAOA(Huang,etal.,2019)inthetransformertotakethecorrelation betweenattentionalqueriesandresultsintoconsideration.”
A Mask-Guided Transformer Network with Topic Token for Remote Sensing Image Captioning
Ren, Z., Gou, S., Guo, Z., Mao, S., & Li, R. (2022) · Remote Sensing · doi:10.3390/rs14122939 · Cited by 38
“To enhance the interactions between sentences and images, an attention on attention (AOA) module has been proposed in AoANet [23].”
Efficient Image Captioning for Edge Devices
Wang, N., Xie, J., Luo, H., Cheng, Q., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25359 · Cited by 39
Pre-gating and contextual attention gate — A new fusion method for multi-modal data tasks
Zhang, D., Nayak, R., & Bashar, M. A. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106553 · Cited by 37
Exploring Pairwise Relationships Adaptively From Linguistic Context in Image Captioning
Zhang, Z., Wu, Q., Wang, Y., & Chen, F. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3093725 · Cited by 34
Image Captioning with Context-Aware Auxiliary Guidance
Song, Z., Zhou, X., Mao, Z., & Tan, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16361 · Cited by 34
“2019) and AoANet (Huang et al. 2019).”
Exploiting Cross-Modal Prediction and Relation Consistency for Semisupervised Image Captioning
Yang, Y., Wei, H., Zhu, H., Yu, D., Xiong, H., & Yang, J. J. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3156367 · Cited by 34
“Details can refer to [9].”
Influential
Metadata
- Authors
- Huang, L., Wang, W., Chen, J., & Wei, X.-Y.
- Affiliations
- Peking University (CN)
- Macau University of Science and Technology (MO)
- Peng Cheng Laboratory (CN)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4633-4642
- OpenAlex
- W2986670728
- Semantic Scholar
- 4c163d4942117179d3e97182e1b280027d7d60a9
- Cited by
- 1,092
- References
- 54
- Citations by year
- 2026: 52 · 2025: 142 · 2024: 197 · 2023: 228 · 2022: 212 · 2021: 188 · 2020: 70 · 2019: 2
- FWCI
- 38.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (U-1613209, 61972217, 61872256)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- image captioning, attention mechanism, attention gate, encoder-decoder architecture, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
