Region-Aware Image Captioning via Interaction Learning
The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Image captioning is one of the primary goals in computer vision which aims to automatically generate natural descriptions for images. Intuitively, human visual system can notice some stimulating regions at first glance, and then volitionally focus on interesting objects within the region. For example, to generate a free-form sentence about “boy-catch-baseball”, the visual region involving “boy” and “baseball” could be first attended and then guide the salient object discovery for the word-by-word generation. Till now, previous captioning works mainly rely on the object-wise modeling and ignore the rich regional patterns. To mitigate the drawback, this paper proposes the region-aware interaction learning method, which aims to explicitly capture the semantic correlations in the region and object dimensions for the word inference. First, given an image, we extract a set of regions which contain diverse objects and their relations. Second, we present the spatial-GCN interaction refining structure which can establish the connection between regions and objects to effectively capture contextual information. Third, we design the dual-attention interaction inference procedure, which enables attention to be calculated in region and object dimensions jointly for the word generation. Specifically, the guidance mechanism is proposed to selectively emphasize semantic inter-dependencies from region to object attentions. Extensive experiments on the MSCOCO dataset demonstrate the superiority…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is not recommended to use the RBN's hidden layers alone to extract features from the data [105-109].”
From Vehicle Detection and Classification via YOLOv8 and Deep Belief Network over Aerial Image Sequences (2023) · cites this paper as [106]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1710.10903 · Cited by 12,107
(2004) · Cited by 9,718
Veličković, P., Casanova, A., Liò, P., Cucurull, G., Romero, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.17863/cam.48429 · Cited by 8,657
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
(2005) · Cited by 4,152
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.131 · Cited by 2,112
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Fang, C., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.503 · Cited by 1,803
Meshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,204
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092
Neural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00611 · Cited by 1,052
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_42 · Cited by 870
BabyTalk: Understanding and Generating Simple Image Descriptions
Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.162 · Cited by 839
Auto-Encoding Scene Graphs for Image Captioning
Yang, X., Tang, K., Zhang, H., & Cai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01094 · Cited by 827
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 733
X-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720
End-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605
Mind's eye: A recurrent visual representation for image caption generation
Chen, X., & Zitnick, C. L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298856 · Cited by 498
Multimodal Transformer With Multi-View Visual Representation for Image Captioning
Yu, J., Li, J., Yu, Z., & Huang, Q. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2947482 · Cited by 468
Semantic Compositional Networks for Visual Captioning
Gan, Z., Gan, C., He, X., Pu, Y., Tran, K., Gao, J., Carin, L., & Deng, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.127 · Cited by 453
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430
Lu, J., Yang, J., Batra, D., & Parikh, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00754 · Cited by 411
Entangled Transformer for Image Captioning
Li, G., Zhu, L., Liu, P., & Yang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00902 · Cited by 375
Corpus-Guided Sentence Generation of Natural Images
Yang, Y., Teo, C. L., Daumé, H., & Aloimonos, Y. (2011) · Cited by 356
Image Captioning with Deep Bidirectional LSTMs
Wang, C., Yang, H., Bartz, C., & Meinel, C. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2964299 · Cited by 287
Recurrent Fusion Network for Image Captioning
Jiang, W., Ma, L., Jiang, Y.-G., Liu, W., & Zhang, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_31 · Cited by 286
Factorizable Net: An Efficient Subgraph-Based Framework for Scene Graph Generation
Li, Y., Ouyang, W., Zhou, B., Shi, J., Zhang, C., & Wang, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_21 · Cited by 283
(2017) · doi:10.1145/2998181.2998364 · Cited by 258
Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Guo, L., Liu, J., Zhu, X., Yao, P., Lu, S., & Lu, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01034 · Cited by 250
Hierarchy Parsing for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00271 · Cited by 206
Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
Gu, J., Cai, J., Wang, G., & Chen, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12266 · Cited by 196
Learning to Count Objects in Natural Images for Visual Question Answering
Zhang, Y., Hare, J. S., & Prügel‐Bennett, A. (2018) · ePrints Soton (University of Southampton) · doi:10.48550/arxiv.1802.05766 · Cited by 177
Self-citationDeep Collaborative Multi-View Hashing for Large-Scale Image Search
Zhu, L., Lu, X., Cheng, Z., Li, J., & Zhang, H. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2974065 · Cited by 170
Language-Conditioned Graph Networks for Relational Reasoning
Hu, R., Rohrbach, A., Darrell, T., & Saenko, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01039 · Cited by 160
Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
Liu, X., Li, H., Shao, J., Chen, D., & Wang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_21 · Cited by 151
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Mao, J., Wei, X., Yang, Y., Wang, J., Huang, Z., & Yuille, A. L. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.291 · Cited by 148
Look Back and Predict Forward in Image Captioning
Qin, Y., Du, J., Zhang, Y., & Lu, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00856 · Cited by 147
Self-citation(2022) · Principles of Data Assimilation · doi:10.1017/9781108924238.008 · Cited by 145
GLA: Global–Local Attention for Image Description
Li, L., Tang, S., Zhang, Y., Deng, L., & Tian, Q. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2751140 · Cited by 140
Self-citationMulti-Level Policy and Reward-Based Deep Reinforcement Learning Framework for Image Captioning
Xu, N., Zhang, H., Liu, A.-A., Nie, W., Su, Y., Nie, J., & Zhang, Y. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2941820 · Cited by 131
Self-citationDual-Stream Recurrent Neural Network for Video Captioning
Xu, N., Liu, A.-A., Wong, Y., Zhang, Y., Nie, W., Su, Y., & Kankanhalli, M. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2867286 · Cited by 120
Self-citationContext-Aware Visual Policy Network for Sequence-Level Image Captioning
Liu, D., Zha, Z.-J., Zhang, H., Zhang, Y., & Wu, F. H. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240632 · Cited by 119
Self-citationReflective Decoding Network for Image Captioning
Ke, L., Pei, W., Li, R., Shen, X., & Tai, Y.-W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00898 · Cited by 109
Learning to Collocate Neural Modules for Image Captioning
Yang, X., Zhang, H., & Cai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00435 · Cited by 96
Action Parsing-Driven Video Summarization Based on Reinforcement Learning
Lei, J., Luan, Q., Song, X., Liu, X., Tao, D., & Song, M. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2860797 · Cited by 85
Shindjalova, R., Prodanova, K., & Svechtarov, V. (2014) · AIP conference proceedings · doi:10.1063/1.4902458 · Cited by 82
Learning to Guide Decoding for Image Captioning
Jiang, W., Ma, L., Chen, X., Zhang, H., & Liu, W. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12283 · Cited by 78
Pointing Novel Objects in Image Captioning
Li, Y., Yao, T., Pan, Y., Chao, H., & Mei, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01278 · Cited by 76
Adaptively Aligned Image Captioning via Adaptive Attention TimePreprint
Huang, L., Wang, W., Xia, Y., & Chen, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09060 · Cited by 76
Self-Critical N-Step Training for Image Captioning
Gao, J., Wang, S., Wang, S., Ma, S., & Gao, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00646 · Cited by 74
Noise Augmented Double-Stream Graph Convolutional Networks for Image Captioning
Wu, L., Xu, M., Sang, L., Yao, T., & Mei, T. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3036860 · Cited by 71
Review Networks for Caption Generation
Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07912 · Cited by 62
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Scene Graph Inference via Multi-Scale Context Modeling
Xu, N., Liu, A.-A., Wong, Y., Nie, W., Su, Y., & Kankanhalli, M. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2990989 · Cited by 39
Self-citation
Cited by
Medical image analysis using deep learning algorithms
Li, M., Jiang, Y., Zhang, Y., & Zhu, H. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1273253 · Cited by 505
Self-citationZhang, B., Shi, H., & Wang, H. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s410301 · Cited by 347
Robust human locomotion and localization activity recognition over multisensory
Khan, D., Alonazi, M., Abdelhaq, M., Mudawi, N. A., Algarni, A. M., Jalal, A., & Liu, H. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1344887 · Cited by 220
Bashir, A. K., Victor, N., Bhattacharya, S., Huynh-The, T., Chengoden, R., Yenduri, G., Maddikunta, P. K. R., Pham, Q.-V., Gadekallu, T. R., & Liyanage, M. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3304790 · Cited by 183
“D lung object to zoom in and rotate for diagnosis) without touching the screen or button-aided control [51].”
BackgroundYenduri, G., Kaluri, R., Rajput, D. S., Lakshmanna, K., Gadekallu, T. R., Mahmud, M., & Brown, D. J. (2023) · IEEE Access · doi:10.1109/access.2023.3289496 · Cited by 95
Metadata
- Authors
- Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y.
- Affiliations
- Tianjin University (CN)
- University of Science and Technology of China (CN)
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2022-06
- Volume
- 32
- Issue
- 6
- Pages
- 3685-3696
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W3195680250
- Cited by
- 128
- References
- 70
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 23 · 2023: 72 · 2022: 14
- FWCI
- 9.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
- Funders
- National Natural Science Foundation of China (61772359, 62002257)
- Grant of Tianjin New Generation Artificial Intelligence Major Program (19ZXZNGX00110, 18ZXZNGX00150)
- China Postdoctoral Science Foundation (2021M692395)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- image captioning, interaction learning, dual attention mechanism, visual attention, context modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.