The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Cross-Modal Attention With Semantic Consistence for Image–Text Matching
Abstract
The task of image-text matching refers to measuring the visual-semantic similarity between an image and a sentence. Recently, the fine-grained matching methods that explore the local alignment between the image regions and the sentence words have shown advance in inferring the image-text correspondence by aggregating pairwise region-word similarity. However, the local alignment is hard to achieve as some important image regions may be inaccurately detected or even missing. Meanwhile, some words with high-level semantics cannot be strictly corresponding to a single-image region. To tackle these problems, we address the importance of exploiting the global semantic consistence between image regions and sentence words as complementary for the local alignment. In this article, we propose a novel hybrid matching approach named Cross-modal Attention with Semantic Consistency (CASC) for image-text matching. The proposed CASC is a joint framework that performs cross-modal attention for local alignment and multilabel prediction for global semantic consistence. It directly extracts semantic labels from available sentence corpus without additional labor cost, which further provides a global similarity constraint for the aggregated region-word similarity obtained by the local alignment. Extensive experiments on Flickr30k and Microsoft COCO (MSCOCO) data sets demonstrate the effectiveness of the proposed CASC on preserving global semantic consistence along with the local alignment and further show its superior image-text matching performance compared with more than 15 state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
(2011) · Cited by 2,752
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · International Conference on Machine Learning · Cited by 1,602
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G. R. G., Levy, R., & Vasconcelos, N. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1873987 · Cited by 1,334
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2587640 · Cited by 875
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779
Self-citationLearning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.541 · Cited by 760
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 739
Dual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.232 · Cited by 726
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 620
Self-citationDual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3383184 · Cited by 589
Self-citationCross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654902 · Cited by 560
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 543
Learning Two-Branch Neural Networks for Image-Text Matching Tasks
Wang, L., Li, Y., Huang, J., & Lazebnik, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2797921 · Cited by 535
Deep Cross-Modal Projection Learning for Image-Text Matching
Zhang, Y., & Lu, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_42 · Cited by 495
Learning Discriminative Binary Codes for Large-scale Cross-modal Retrieval
Xu, X., Shen, F., Yang, Y., Shen, H. T., & Li, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2676345 · Cited by 458
Self-citationUnsupervised Deep Hashing with Similarity-Adaptive and Discrete Optimization
Shen, F., Xu, Y., Liu, L., Yang, Y., Huang, Z., & Shen, H. T. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2789887 · Cited by 407
Self-citationTriplet-Based Deep Hashing Network for Cross-Modal Retrieval
Deng, C., Chen, Z., Liu, X., Gao, X., & Tao, D. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2821921 · Cited by 400
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00750 · Cited by 396
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00645 · Cited by 342
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334
CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
Peng, Y., & Qi, J. (2019) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3284750 · Cited by 290
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R., Saenko, K., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.8 · Cited by 286
Associating neural word embeddings with deep image representations using Fisher Vectors
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299073 · Cited by 281
Identity-Aware Textual-Visual Matching with Latent Co-attention
Li, S., Xiao, T., Li, H., Yang, W., & Wang, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.209 · Cited by 277
Self-Supervised Video Hashing With Hierarchical Binary Auto-Encoder
Song, J., Zhang, H., Li, X., Gao, L., Wang, M., & Hong, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2814344 · Cited by 276
Describing Video With Attention-Based Bidirectional LSTM
Bin, Y., Yang, Y., Shen, F., Xie, N., Shen, H. T., & Li, X. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2831447 · Cited by 270
Self-citationInstance-Aware Image and Sentence Matching with Selective Multimodal LSTM
Huang, Y., Wang, W., & Wang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.767 · Cited by 233
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228
Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question AnsweringPreprint
Kazemi, V., & Elqursh, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03162 · Cited by 210
Deep adversarial metric learning for cross-modal retrieval
Xu, X., He, L., Lu, H., Gao, L., & Ji, Y. (2019) · World Wide Web · doi:10.1007/s11280-018-0541-x · Cited by 210
Self-citationTernary Adversarial Networks With Self-Supervision for Zero-Shot Cross-Modal Retrieval
Xu, X., Lu, H., Song, J., Yang, Y., Shen, H. T., & Li, X. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2928180 · Cited by 205
Self-citationVSE++: Improving Visual-Semantic Embeddings with Hard Negatives.
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2018) · Cited by 184
Linking Image and Text with 2-Way Nets
Eisenschtat, A., & Wolf, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.201 · Cited by 171
R²GAN: Cross-Modal Recipe Retrieval With Generative Adversarial Network
Zhu, B., Ngo, C., Chen, J., & Hao, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01174 · Cited by 170
RNN Fisher Vectors for Action Recognition and Image Annotation
Lev, G., Sadeh, G., Klein, B., & Wolf, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_50 · Cited by 164
Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding
Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.208 · Cited by 161
Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
Wang, T., Xu, X., Yang, Y., Hanjalic, A., Shen, H. T., & Song, J. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350875 · Cited by 161
Self-citationLearning a Recurrent Residual Fusion Network for Multimodal Matching
Liu, Y., Guo, Y., Bakker, E. M., & Lew, M. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.442 · Cited by 159
Shared Predictive Cross-Modal Deep Quantization
Yang, E., Deng, C. D., Li, C., Liu, W., Li, J., & Tao, D. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2793863 · Cited by 158
Graph Convolutional Network Hashing
Zhou, X. S., Shen, F., Liu, L., Liu, W., Nie, L., Yang, Y., & Shen, H. T. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2883970 · Cited by 148
Self-citationAdversarial Examples for Hamming Space Search
Yang, E., Liu, T., Deng, C. D., & Tao, D. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2882908 · Cited by 130
Bi-Directional Spatial-Semantic Attention Networks for Image-Text Matching
Huang, F., Zhang, X., Zhao, Z., & Li, Z. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2882225 · Cited by 107
Hashing with Angular Reconstructive Embeddings
Hu, M., Yang, Y., Shen, F., Xie, N., & Shen, H. T. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2749147 · Cited by 103
Self-citationDeep Semantic-Preserving Ordinal Hashing for Cross-Modal Similarity Search
Jin, L., Li, K., Li, Z., Xiao, F., Qi, G.-J., & Tang, J. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2869601 · Cited by 80
Hetero-Manifold Regularisation for Cross-Modal Hashing
Zheng, F., Tang, Y., & Shao, L. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2645565 · Cited by 79
Joint Global and Co-Attentive Representation Learning for Image-Sentence Retrieval
Wang, S., Chen, Y., Zhuo, J., Huang, Q., & Tian, Q. C. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240535 · Cited by 59
CycleMatch: A cycle-consistent embedding network for image-text matching
Liu, Y., Guo, Y., Liu, L., Bakker, E. M., & Lew, M. S. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.05.008 · Cited by 46
Cross-modal Bidirectional Translation via Reinforcement Learning
Qi, J., & Peng, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/365 · Cited by 42
Bidirectional image-sentence retrieval by local and global deep matching
Ma, L., Jiang, W., Jie, Z., & Wang, X. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.089 · Cited by 35
Cross-media Multi-level Alignment with Relation Attention Network
Qi, J., Peng, Y., & Yuan, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/124 · Cited by 31
Cited by
Deep Fuzzy Hashing Network for Efficient Image Retrieval
Lu, H., Zhang, M., Xu, X., Li, Y., & Shen, H. T. (2021) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2020.2984991 · Cited by 340
“, image classification [13], [14], object detection [15], image segmentation [16], image-text matching [17], etc.”
BackgroundSelf-citationSGUIE-Net: Semantic Attention Guided Underwater Image Enhancement With Multi-Scale Perception
Qi, Q., Li, K., Zheng, H., Gao, X., Hou, G., & Sun, K. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3216208 · Cited by 279
Metadata
- Authors
- Xu, X., Wang, T., Yang, Y., Zuo, L., Shen, F., & Shen, H. T.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2020-12
- Volume
- 31
- Issue
- 12
- Pages
- 5412-5425
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3005971801
- Cited by
- 279
- References
- 63
- FWCI
- 16.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.