The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Dual-path Convolutional Image-Text Embeddings with Instance Loss
Open accessAbstract
Matching images and sentences demands a fine understanding of both modalities. In this article, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image/text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss on heterogeneous features (i.e., text and image features) is less effective, because it is hard to find appropriate triplets at the beginning. So the naive way of using the ranking loss may compromise the network from learning inter-modal relationship. To address this problem, we propose the instance loss, which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image/text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this article constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language-based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,202
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850
DeepReID: Deep Filter Pairing Neural Network for Person Re-identification
Li, W., Zhao, R., Xiao, T., & Wang, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.27 · Cited by 2,611
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G. R. G., Levy, R., & Vasconcelos, N. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1873987 · Cited by 1,334
Part-Based R-CNNs for Fine-Grained Category Detection
Zhang, N., Donahue, J., Girshick, R., & Darrell, T. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_54 · Cited by 1,150
Very Deep Convolutional Networks for Text Classification
Conneau, A., Schwenk, H., Barrault, L., & Lecun, Y. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1104 · Cited by 1,046
Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks
Chen, Y., Xu, L., Liu, K., Zeng, D., & Zhao, J. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1017 · Cited by 906
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2587640 · Cited by 875
A Discriminatively Learned CNN Embedding for Person Reidentification
Zheng, Z., Zheng, L., & Yang, Y. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3159171 · Cited by 874
Self-citationImproving person re-identification by attribute and identity learning
Lin, Y., Zheng, L., Zheng, Z., Wu, Y., Hu, Z., Yan, C., & Yang, Y. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.06.006 · Cited by 858
Self-citationLearning Deep Representations of Fine-Grained Visual Descriptions
Reed, S., Akata, Z., Lee, H., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.13 · Cited by 760
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.541 · Cited by 760
Dual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.232 · Cited by 726
Unsupervised Person Re-identification: Clustering and Fine-tuning
Fan, H., Zheng, L., Yan, C., & Yang, Y. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3243316 · Cited by 636
Self-citationPerson Search with Natural Language Description
Li, S., Xiao, T., Li, H., Zhou, B., Yue, D., & Wang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.551 · Cited by 550
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00750 · Cited by 396
Deep correlation for matching images and text
Yan, F., & Mikolajczyk, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298966 · Cited by 395
Cross-Modal Retrieval With CNN Visual Features: A New Baseline
Wei, Y., Zhao, Y., Lu, C., Wei, S., Liu, L., Zhu, Z., & Yan, S. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2519449 · Cited by 370
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00645 · Cited by 342
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334
Multi-scale Deep Learning Architectures for Person Re-identification
Qian, X., Fu, Y., Jiang, Y.-G., Xiang, T., & Xue, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.577 · Cited by 319
Associating neural word embeddings with deep image representations using Fisher Vectors
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299073 · Cited by 281
Identity-Aware Textual-Visual Matching with Latent Co-attention
Li, S., Xiao, T., Li, H., Yang, W., & Wang, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.209 · Cited by 277
Learning Coupled Feature Spaces for Cross-Modal Matching
Wang, K., He, R., Wang, W., Wang, L., & Tan, T. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.261 · Cited by 235
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM
Huang, Y., Wang, W., & Wang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.767 · Cited by 233
Large-Scale Cross-Modality Search via Collective Matrix Factorization Hashing
Ding, G., Guo, Y., Zhou, J., & Gao, Y. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2607421 · Cited by 218
Progressive Learning for Person Re-Identification With One Example
Wu, Y., Lin, Y., Dong, X., Yan, Y., Bian, W., & Yang, Y. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2891895 · Cited by 211
Self-citationLearning Aligned Cross-Modal Representations from Weakly Aligned Data
Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.321 · Cited by 188
Uncovering the Temporal Context for Video Question Answering
Zhu, L., Xu, Z., Yang, Y., & Hauptmann, A. G. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1033-7 · Cited by 181
Self-citationLinking Image and Text with 2-Way Nets
Eisenschtat, A., & Wolf, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.201 · Cited by 171
Unsupervised Semantic-Preserving Adversarial Hashing for Image Search
Deng, C., Yang, E., Liu, T., Li, J., Liu, W., & Tao, D. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2903661 · Cited by 167
RNN Fisher Vectors for Action Recognition and Image Annotation
Lev, G., Sadeh, G., Klein, B., & Wolf, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_50 · Cited by 164
Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding
Niu, Z., Zhou, M., Wang, L., Gao, X., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.208 · Cited by 161
Image Captioning with Deep Bidirectional LSTMs and Multi-Task Learning
Wang, C., Yang, H., & Meinel, C. (2018) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3115432 · Cited by 153
Effective multi-modal retrieval based on stacked auto-encoders
Wang, W., Ooi, B. C., Yang, X., Zhang, D., & Zhuang, Y. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2732296.2732301 · Cited by 143
Image Classification by Cross-Media Active Learning With Privileged Information
Yan, Y., Nie, F., Li, W., Gao, C., Yang, Y. P., & Xu, D. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2602938 · Cited by 141
Self-citationPerson Reidentification via Structural Deep Metric Learning
Yang, X., Zhou, P., & Wang, M. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2861991 · Cited by 140
Flexible Multi-View Dimensionality Co-Reduction
Zhang, C., Fu, H., Hu, Q., Zhu, P., & Cao, X. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2627806 · Cited by 138
Cross-Modal Retrieval via Deep and Bidirectional Representation Learning
He, Y., Xiang, S., Kang, C., Wang, J., & Pan, C. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2558463 · Cited by 137
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
Cornia, M., Baraldi, L., Serra, G., & Cucchiara, R. (2018) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3177745 · Cited by 114
Discriminative Dictionary Learning With Common Label Alignment for Cross-Modal Retrieval
Deng, C. D., Tang, X., Yan, J., Liu, W., & Gao, X. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2508146 · Cited by 99
Framing image description as a ranking task
HodoshMicah, YoungPeter, & HockenmaierJulia (2013) · doi:10.5555/2566972.2566993 · Cited by 98
Multimodal Discriminative Binary Embedding for Large-Scale Cross-Modal Retrieval
Wang, D., Gao, X., Wang, X., He, L., & Yuan, B. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2592800 · Cited by 93
Modality-Dependent Cross-Media Retrieval
Wei, Y., Zhao, Y., Zhu, Z., Wei, S., Xiao, Y., Feng, J., & Yan, S. (2016) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2775109 · Cited by 90
Cross-media semantic representation via bi-directional learning to rank
Wu, F. Q., Lu, X., Zhang, Z. M., et al. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502097 · Cited by 82
Leveraging Visual Question Answering for Image-Caption Ranking
Lin, X., & Parikh, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_17 · Cited by 82
Image Caption Generation with Part of Speech Guidance
He, X., Shi, B., Bai, X., Xia, G.-S., Zhang, Z., & Dong, W. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.10.018 · Cited by 79
Query-Adaptive Hash Code Ranking for Large-Scale Multi-View Visual Search
Liu, X., Huang, L., Deng, C. D., Lang, B., & Tao, D. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2593344 · Cited by 74
Cross-Modal Subspace Learning via Pairwise Constraints
He, R., Zhang, M., Wang, L., et al. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2466106 · Cited by 73
Twitter100k: A Real-World Dataset for Weakly Supervised Cross-Media Retrieval
Hu, Y., Zheng, L., Yang, Y., & Huang, Y. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2760101 · Cited by 71
Self-citationDiscriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries
Zhang, Y., Yuan, L., Guo, Y., He, Z., Huang, I.-A., & Lee, H. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.122 · Cited by 51
Modality-Invariant Image-Text Embedding for Image-Sentence Matching
Liu, R., Zhao, Y., Wei, S., Zheng, L., & Yang, Y. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3300939 · Cited by 38
Self-citationCorrespondence Autoencoders for Cross-Modal Retrieval
Feng, F., Wang, X., Li, R., & Ahmad, I. (2015) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/2808205 · Cited by 32
Cited by
VinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00553 · Cited by 907
Multi-Modality Cross Attention Network for Image and Sentence Matching
Wei, X., Zhang, T., Li, Y., Zhang, Y., & Wu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01095 · Cited by 451
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., & Han, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01267 · Cited by 429
University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization
Zheng, Z., Wei, Y., & Yang, Y. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413896 · Cited by 402
Self-citationCLIP-Driven Fine-Grained Text-Image Person Re-Identification
Yan, S., Dong, N., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3327924 · Cited by 330
Metadata
- Authors
- Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D.
- Venue
- ACM Transactions on Multimedia Computing, Communications, and Applications
- Published
- 2020-05-22
- Volume
- 16
- Issue
- 2
- Pages
- 1-23
- ISSN
- 1551-6857 (ISSN-L)
- 1551-6865
- OpenAlex
- W2883311563
- Cited by
- 589
- References
- 65
- FWCI
- 28.52
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.