The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Abstract
Text-to-image person retrieval aims to identify the target person based on a given textual description query. The primary challenge is to learn the mapping of visual and textual modalities into a common latent space. Prior works have attempted to address this challenge by leveraging separately pre-trained unimodal models to extract visual and textual features. However, these approaches lack the necessary underlying alignment capabilities required to match multimodal data effectively. Besides, these works use prior information to explore explicit part alignments, which may lead to the distortion of intra-modality information. To alleviate these issues, we present IRRA: a cross-modal Implicit Relation Reasoning and Aligning framework that learns relations between local visual-textual tokens and enhances global image-text matching without requiring additional prior supervision. Specifically, we first design an Implicit Relation Reasoning module in a masked language modeling paradigm. This achieves cross-modal interaction by integrating the visual cues into the textual tokens with a cross-modal multimodal interaction encoder. Secondly, to globally align the visual and textual embeddings, Similarity Distribution Matching is proposed to minimize the KL divergence between image-text similarity distributions and the normalized label matching distributions. The proposed method achieves new state-of-the-art results on all three public datasets, with a notable margin of about 3%-9% for Rank-1 accuracy compared to prior methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 1×
“1 2 and 3 denotes different CLIP models fine-tuned with the original CLIP loss (InfoNCE [34]).”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
“Cloze Procedure”: A New Tool for Measuring Readability
Taylor, W. L. (1953) · Journalism Quarterly · doi:10.1177/107769905303000401 · Cited by 2,332
Deep Learning for Person Re-Identification: A Survey and Outlook
Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., & Hoi, S. C. H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054775 · Cited by 2,259
Self-citationViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Bag of Tricks and a Strong Baseline for Deep Person Re-Identification
Luo, H., Gu, Y., Liao, X., Lai, S., & Jiang, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00190 · Cited by 1,584
TransReID: Transformer-based Object Re-Identification
He, S., Luo, H., Wang, P., Wang, F., Li, H., & Jiang, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01474 · Cited by 1,244
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Kim, W., Son, B., & Kim, I. (2021) · arXiv · doi:10.48550/arxiv.2102.03334 · Cited by 951 · cited 1×
“In singlestream models [6, 23, 41], text and visual features are concatenated and then fed into a single transformer encoder.”
MethodVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Dual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3383184 · Cited by 589
Person Search with Natural Language Description
Li, S., Xiao, T., Li, H., Zhou, B., Yue, D., & Wang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.551 · Cited by 550
Deep Cross-Modal Projection Learning for Image-Text Matching
Zhang, Y., & Lu, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_42 · Cited by 495
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01763 · Cited by 348
CLIP-Driven Fine-Grained Text-Image Person Re-Identification
Yan, S., Dong, N., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3327924 · Cited by 330
Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identificationPreprint
Ding, Z., Ding, C., Shao, Z., & Tao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12666 · Cited by 286
DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval
Zhu, A., Wang, Z., Li, Y., Wan, X., Jin, J., Wang, T., Hu, F., & Hua, G. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475369 · Cited by 278
Identity-Aware Textual-Visual Matching with Latent Co-attention
Li, S., Xiao, T., Li, H., Yang, W., & Wang, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.209 · Cited by 277
See Finer, See More: Implicit Modality Alignment for Text-Based Person Retrieval
Shu, X., Wen, W. X., Wu, H., Chen, K., Song, Y., Qiao, R., Ren, B., & Wang, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25072-9_42 · Cited by 261
Adversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 250
TIPCB: A simple but effective part-based convolutional baseline for text-based person search
Chen, Y., Zhang, G., Lu, Y., Wang, Z., & Zheng, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.081 · Cited by 237
ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
Wang, Z., Fang, Z., Wang, J., & Yang, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_24 · Cited by 232
Image-Specific Information Suppression and Implicit Local Alignment for Text-Based Person Search
Yan, S., Tang, H., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3310118 · Cited by 200
AXM-Net: Implicit Cross-Modal Feature Alignment for Person Re-identification
Farooq, A., Awais, M., Kittler, J., & Khalid, S. S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20370 · Cited by 195
Learning Granularity-Unified Representations for Text-to-Image Person Re-identification
Shao, Z., Zhang, X., Fang, M., Lin, Z., Wang, J., & Ding, C. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548028 · Cited by 195
NFormer: Robust Person Re-identification with Neighbor Transformer
Wang, H., Shen, J., Liu, Y., Gao, Y., & Gavves, E. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00715 · Cited by 194
Pose-Guided Multi-Granularity Attention Network for Text-Based Person Search
Jing, Y., Si, C., Wang, J., Wang, W., Wang, L., & Tan, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6777 · Cited by 179
Text-Based Person Search with Limited Data
Han, X., He, S., Zhang, L., & Xiang, T. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.10 · Cited by 173
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149
Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person SearchPreprint
Gao, C., Cai, G., Jiang, X., Zheng, F., Zhang, J., Gong, Y., Peng, P., Guo, X., & Sun, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03036 · Cited by 149
Improving Text-Based Person Search by Spatial Matching and Adaptive Threshold
Chen, T., Xu, C., & Luo, J. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00208 · Cited by 137
Wang, Z., Zhu, A., Xue, J., Wan, X., Liu, C., Wang, T., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548166 · Cited by 133
CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval
Wang, Z., Zhu, A., Xue, J., Wan, X., Liu, C., Wang, T., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548057 · Cited by 131
Learning Semantic-Aligned Feature Representation for Text-Based Person Search
Li, S., Cao, M., & Zhang, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746846 · Cited by 121
LapsCore: Language-guided Person Search via Color Reasoning
Wu, Y., Yan, Z., Han, X., Li, G., Zou, C., & Cui, S. R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00165 · Cited by 119
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Sun, S., Chen, Y.-C., Li, L., Wang, S., Fang, Y., & Liu, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.77 · Cited by 111
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100
Cross-Modal Knowledge Adaptation for Language-Based Person Search
Chen, Y., Huang, R., Chang, H., Tan, C., Xue, T., & Ma, B. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3068825 · Cited by 96
Contextual Representation Learning beyond Masked Language Modeling
Fu, Z., Zhou, W., Xu, J., Zhou, H., & Li, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.193 · Cited by 17
Metadata
- Authors
- Jiang, D., & Ye, M.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 2787-2797
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386065462
- Cited by
- 323
- References
- 54
- FWCI
- 36.73
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.