The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
CLIP-Driven Fine-Grained Text-Image Person Re-Identification
Open accessAbstract
Text-Image Person Re-identification (TIReID) aims to retrieve the image corresponding to the given text query from a pool of candidate images. Existing methods employ prior knowledge from single-modality pre-training to facilitate learning, but lack multi-modal correspondence information. Vision-Language Pre-training, such as CLIP (Contrastive Language-Image Pretraining), can address the limitation. However, CLIP falls short in capturing fine-grained information, thereby not fully leveraging its powerful capacity in TIReID. Besides, the popular explicit local matching paradigm for mining fine-grained information heavily relies on the quality of local parts and cross-modal inter-part interaction/guidance, leading to intra-modal information distortion and ambiguity problems. Accordingly, in this paper, we propose a CLIP-driven Fine-grained information excavation framework (CFine) to fully utilize the powerful knowledge of CLIP for TIReID. To transfer the multi-modal knowledge effectively, we conduct fine-grained information excavation to mine modality-shared discriminative details for global alignment. Specifically, we propose a multi-level global feature learning (MGF) module that fully mines the discriminative local information within each modality, thereby emphasizing identity-related discriminative clues through enhanced interaction between global image (text) and informative local patches (words). MGF generates a set of enhanced global features for later inference. Furthermore, we design cross-grained feature refinement (CFR) and fine-grained correspondence discovery (FCD) modules to establish cross-modal correspondence at both coarse and fine-grained levels (image-word, sentence-patch, word-patch), ensuring the reliability of informative local patches/words. CFR and FCD are removed during inference to optimize computational efficiency. Extensive experiments on multiple benchmarks demonstrate the superior performance of our method in TIReID.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Beyond Part Models: Person Retrieval with Refined Part Pooling (and A Strong Convolutional Baseline)
Sun, Y., Zheng, L., Yang, Y., Tian, Q., & Wang, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_30 · Cited by 2,522
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · cited 2×
“Recently, many works [16], [33], [34], [35], [36], [37] have sought to extend the concept of pre-training to the multi-modal domain, known as visual-language pre-training (VLP), leading to significant progress.”
BackgroundMethodAlign before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078
CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning
Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., & Li, T. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.07.028 · Cited by 762
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., & Lu, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01755 · Cited by 631
Dual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3383184 · Cited by 589
Pose-Invariant Embedding for Deep Person Re-Identification
Zheng, L., Huang, Y., Lu, H., & Yang, Y. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2910414 · Cited by 583
Person Search with Natural Language Description
Li, S., Xiao, T., Li, H., Zhou, B., Yue, D., & Wang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.551 · Cited by 550
Deep Cross-Modal Projection Learning for Image-Text Matching
Zhang, Y., & Lu, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_42 · Cited by 495
PointCLIP: Point Cloud Understanding by CLIP
Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, Y., Gao, P., & Li, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00836 · Cited by 442
CRIS: CLIP-Driven Referring Image Segmentation
Wang, Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., & Liu, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01139 · Cited by 404
Learning attention-guided pyramidal features for few-shot fine-grained recognition
Tang, H., Yuan, C., Li, Z., & Tang, J. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108792 · Cited by 399
Self-citationCLIP2Video: Mastering Video-Text Retrieval via Image CLIPPreprint
Fang, H., Xiong, P., Xu, L., & Chen, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11097 · Cited by 398
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., & Ling, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19772-7_1 · Cited by 326
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval
Ma, Y., Xu, G., Sun, X., Yan, M., Zhang, J., & Ji, R. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547910 · Cited by 321
Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identificationPreprint
Ding, Z., Ding, C., Shao, Z., & Tao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12666 · Cited by 286
DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval
Zhu, A., Wang, Z., Li, Y., Wan, X., Jin, J., Wang, T., Hu, F., & Hua, G. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475369 · Cited by 278
Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification
Zhu, H., Ke, W., Li, D., Liu, J., Tian, L., & Shan, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00465 · Cited by 277
See Finer, See More: Implicit Modality Alignment for Text-Based Person Retrieval
Shu, X., Wen, W. X., Wu, H., Chen, K., Song, Y., Qiao, R., Ren, B., & Wang, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25072-9_42 · Cited by 261
Adversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 250
Improving Description-Based Person Re-Identification by Multi-Granularity Image-Text Alignments
Niu, K., Huang, Y., Ouyang, W., & Wang, L. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2984883 · Cited by 238
TIPCB: A simple but effective part-based convolutional baseline for text-based person search
Chen, Y., Zhang, G., Lu, Y., Wang, Z., & Zheng, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.081 · Cited by 237
TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval
Liu, Y., Xiong, P., Xu, L., Cao, S., & Jin, Q. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19781-9_19 · Cited by 234
ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
Wang, Z., Fang, Z., Wang, J., & Yang, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_24 · Cited by 232
Image-Specific Information Suppression and Implicit Local Alignment for Text-Based Person Search
Yan, S., Tang, H., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3310118 · Cited by 200
Self-citationAXM-Net: Implicit Cross-Modal Feature Alignment for Person Re-identification
Farooq, A., Awais, M., Kittler, J., & Khalid, S. S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20370 · Cited by 195
Learning Granularity-Unified Representations for Text-to-Image Person Re-identification
Shao, Z., Zhang, X., Fang, M., Lin, Z., Wang, J., & Ding, C. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548028 · Cited by 195
Pose-Guided Multi-Granularity Attention Network for Text-Based Person Search
Jing, Y., Si, C., Wang, J., Wang, W., Wang, L., & Tan, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6777 · Cited by 179
Chen, D., Li, H., Liu, X., Shen, Y., Shao, J., Yuan, Z., & Wang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_4 · Cited by 162
Boosting Few-Shot Fine-Grained Recognition With Background Suppression and Foreground Alignment
Zha, Z., Tang, H., Sun, Y., & Tang, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3236636 · Cited by 160
Self-citationCLIP4Caption: CLIP for Video Caption
Tang, M., Wang, Z., Liu, Z., Rao, F., Li, D., & Li, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3479207 · Cited by 156
Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person SearchPreprint
Gao, C., Cai, G., Jiang, X., Zheng, F., Zhang, J., Gong, Y., Peng, P., Guo, X., & Sun, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03036 · Cited by 149
Knowledge-Guided Semantic Transfer Network for Few-Shot Image Recognition
Li, Z., Tang, H., Peng, Z., Qi, G.-J., & Tang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3240195 · Cited by 140
Self-citationWang, Z., Zhu, A., Xue, J., Wan, X., Liu, C., Wang, T., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548166 · Cited by 133
CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval
Wang, Z., Zhu, A., Xue, J., Wan, X., Liu, C., Wang, T., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548057 · Cited by 131
Text-based Person Search via Attribute-aided Matching
Aggarwal, S., Babu, R. V., & Chakraborty, A. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093640 · Cited by 129
Learning Semantic-Aligned Feature Representation for Text-Based Person Search
Li, S., Cao, M., & Zhang, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746846 · Cited by 121
LapsCore: Language-guided Person Search via Color Reasoning
Wu, Y., Yan, Z., Han, X., Li, G., Zou, C., & Cui, S. R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00165 · Cited by 119
Progressive learning with multi-scale attention network for cross-domain vehicle re-identification
Wang, Y., Peng, J., Wang, H., et al. (2022) · Science China Information Sciences · doi:10.1007/s11432-021-3383-y · Cited by 116
Deep Adversarial Graph Attention Convolution Network for Text-Based Person Search
Liu, J., Zha, Z.-J., Hong, R., Wang, M., & Zhang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350991 · Cited by 114
Hierarchical Gumbel Attention Network for Text-based Person Search
Zheng, K., Liu, W., Liu, J., Zha, Z.-J., & Mei, T. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413864 · Cited by 100
Text-based Person Search via Multi-Granularity Embedding Learning
Wang, C., Luo, Z., Lin, Y., & Li, S. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/148 · Cited by 97
Cross-Modal Knowledge Adaptation for Language-Based Person Search
Chen, Y., Huang, R., Chang, H., Tan, C., Xue, T., & Ma, B. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3068825 · Cited by 96
LocVTP: Video-Text Pre-training for Temporal Localization
Cao, M., Yang, T., Weng, J., Zhang, C., Wang, J., & Zou, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_3 · Cited by 77
Zhang, L., Du, G., Liu, F., et al. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3085978 · Cited by 74
Self-citationAttribute-Identity Embedding and Self-Supervised Learning for Scalable Person Re-Identification
Li, H., Yan, S., Yu, Z., & Tao, D. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2952550 · Cited by 68
Self-citationSUM: Serialized Updating and Matching for text-based person retrieval
Wang, Z., Zhu, A., Xue, J., Jiang, D., Liu, C., Li, Y., & Hu, F. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108891 · Cited by 50
Cross-Modal Retrieval with Heterogeneous Graph Embedding
Chen, D., Wang, M., Chen, H., Wu, L. Y., Qin, J., & Peng, W. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548195 · Cited by 47
Language Person Search with Mutually Connected Classification Loss
Wang, Y., Bo, C., Wang, D., Wang, S., Qi, Y., & Lu, H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682456 · Cited by 43
Textual Dependency Embedding for Person Search by Language
Niu, K., Huang, Y., & Wang, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413895 · Cited by 43
AMEN: Adversarial Multi-space Embedding Network for Text-Based Person Re-identification
Wang, Z., Xue, J., Zhu, A., Li, Y., Zhang, M., & Zhong, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88007-1_38 · Cited by 27
Yan, S., Zhang, Y., Xie, M., Zhang, D., & Yu, Z. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.09.054 · Cited by 21
Self-citationLaT: Latent Translation with Cycle-Consistency for Video-Text RetrievalPreprint
Bai, J., Liu, C., Ni, F., Wang, H., Hu, M., Guo, X., & Cheng, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.04858 · Cited by 16
Cited by
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Jiang, D., & Ye, M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00273 · Cited by 323
Metadata
- Authors
- Yan, S., Dong, N., Zhang, L., & Tang, J.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2023
- Volume
- 32
- Pages
- 6032-6046
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W4388145471
- Cited by
- 330
- References
- 60
- FWCI
- 30.00
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.