The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Visual Semantic Reasoning for Image-Text Matching
Open accessAbstract
Image-text matching has been a hot research topic bridging the vision and language areas. It remains challenging because the current representation of image usually lacks global semantic concepts as in its corresponding text caption. To address this issue, we propose a simple and interpretable reasoning model to generate visual representation that captures key objects and semantic concepts of a scene. Specifically, we first build up connections between image regions and perform reasoning with Graph Convolutional Networks to generate features with semantic relationships. Then, we propose to use the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually generate the representation for the whole scene. Experiments validate that our method achieves a new state-of-the-art for the image-text matching on MS-COCO and Flickr30K datasets. It outperforms the current best method by 6.8% relatively for image retrieval and 4.8% relatively for caption retrieval on MS-COCO (Recall@1 using 1K test set). On Flickr30K, our model improves image retrieval by 12.6% relatively and caption retrieval by 5.8% relatively (Recall@1).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
Newell, A. (1980) · Cognitive Science · doi:10.1207/s15516709cog0402_2 · Cited by 1,458
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017) · arXiv · doi:10.48550/arxiv.1706.01427 · Cited by 1,281 · cited 1×
“Inspired by recent advances in deep learning based visual reasoning [2, 35, 42], we build up a region relationship reasoning model to enhance the region-based representation by considering the semantic correlation between image regions.”
MethodHobbs, J. R., Stickel, M. E., Appelt, D. E., & Martin, P. (1993) · Artificial Intelligence · doi:10.1016/0004-3702(93)90015-4 · Cited by 1,223
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
Temporal Relational Reasoning in Videos
Zhou, B., Andonian, A., Oliva, A., & Torralba, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_49 · Cited by 1,067
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_42 · Cited by 870
Self-citationVSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 543
Tell Me Where to Look: Guided Attention Inference Network
Li, K., Wu, Z., Peng, K.-C., Ernst, J., & Fu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00960 · Cited by 535
Self-citationBottom-Up and Top-Down Attention: Different Processes and Overlapping Neural Systems
Katsuki, F., & Constantinidis, C. (2014) · The Neuroscientist · doi:10.1177/1073858413514136 · Cited by 505
Random Walk Inference and Learning in A Large Scale Knowledge Base
Lao, N., Mitchell, T. M., & Cohen, W. W. (2018) · Figshare · doi:10.1184/r1/6476267 · Cited by 482
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00750 · Cited by 396
Visual Semantic Navigation using Scene PriorsPreprint
Yang, W. P., Wang, X., Farhadi, A., Gupta, A., & Mottaghi, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06543 · Cited by 352
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00645 · Cited by 342
Scene Graph Generation With External Knowledge and Image Reconstruction
Gu, J., Zhao, H., Lin, Z., Li, S., Cai, J., & Ling, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00207 · Cited by 308
Order-Embeddings of Images and Language
Vendrov, I., Kiros, R., Fidler, S., & Urtasun, R. (2015) · arXiv · doi:10.48550/arxiv.1511.06361 · Cited by 296
Associating neural word embeddings with deep image representations using Fisher Vectors
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299073 · Cited by 281
Iterative Visual Reasoning Beyond Convolutions
Chen, X., Li, L.-J., Fei-Fei, L., & Gupta, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00756 · Cited by 236
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM
Huang, Y., Wang, W., & Wang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.767 · Cited by 233
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives.
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2018) · Cited by 184
Linking Image and Text with 2-Way Nets
Eisenschtat, A., & Wolf, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.201 · Cited by 171
Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding
Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.208 · Cited by 161
Learning a Recurrent Residual Fusion Network for Multimodal Matching
Liu, Y., Guo, Y., Bakker, E. M., & Lew, M. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.442 · Cited by 159
Multimodal Style Transfer via Graph Cuts
Zhang, Y., Fang, C., Wang, Y., Wang, Z., Lin, Z., Fu, Y., & Yang, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00604 · Cited by 98
Self-citationLeveraging Visual Question Answering for Image-Caption Ranking
Lin, X., & Parikh, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_17 · Cited by 89
Guided Attention Inference Network
Li, K., Wu, Z., Peng, K., Ernst, J., & Fu, Y. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2921543 · Cited by 61
Self-citationSupport Neighbor Loss for Person Re-Identification
Kai, L., Ding, Z., Li, K., Zhang, Y., & Fu, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240674 · Cited by 34
Self-citationAttention Bridging Network for Knowledge Transfer
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00530 · Cited by 28
Self-citationChen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Cited by
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
“R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Zero-shot (Faghri et al., 2018), VSRN (Li et al., 2019), DE I2T (Parekh et al., 2021), and DE T2T+I2T (Parekh et al., 2021). image R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 VSE++ ERNIE-ViL (Yu et al., 2020), VILLA (Gan et al., 2020) and Oscar (Li et al., 20…”
BackgroundMethodInfluentialAlign before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078
Huang, S.-C., Shen, L., Lungren, M. P., & Yeung, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00391 · Cited by 489
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., & Han, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01267 · Cited by 429
Similarity Reasoning and Filtration for Image-Text Matching
Diao, H., Zhang, Y., Ma, L., & Lu, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16209 · Cited by 361
Self-citation
Metadata
- Authors
- Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Volume
- 2019
- Pages
- 4653-4661
- OpenAlex
- W2988823324
- Cited by
- 599
- References
- 44
- FWCI
- 20.32
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.