COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation
Abstract
There has been a recent surge of interest in cross-modal pre-training. However, existed approaches pre-train a one-stream model to learn joint vision-language representation, which suffers from calculation explosion when conducting cross-modal retrieval. In this work, we propose the Contrastive Cross-Modal Knowledge Sharing Pretraining (COOKIE) method to learn universal text-image representations. There are two key designs in it, one is the weight-sharing transformer on top of the visual and textual encoders to align text and image semantically, the other is three kinds of contrastive learning designed for sharing knowledge between different modalities. Cross-modal knowledge sharing greatly promotes the learning of unimodal representation. Experiments on multi-modal matching tasks including cross-modal retrieval, text matching, and image retrieval show the effectiveness and efficiency of our pre-training framework. Our COOKIE finetuned on cross-modal datasets MSCOCO, Flickr30K, and MSRVTT achieves new state-of-the-art results while using only 3/1000 inference time comparing to one-stream models. There are also 5.7% and 3.9% improvements in the task of image retrieval and text matching. Source code will be available at https://github.com/kywen1119/COOKIE.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Processing the visual input in convolution-free manner simplifies the pipeline, e.g., E2E-VLP and ViLT.
“One simplifying is processing the visual input in convolution-free manner, e.g., E2E-VLP [271], ViLT [192].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [271]
Recent remedies for fine-grained semantic alignment include random sampling, learning a concept dictionary, jointly learning a region detector, and representation aligning before mask prediction.
“Several remedies introduced recently include random sampling [113], learning concept dictionary [203], jointly learning a region detector [271], and representation aligning before mask prediction [263].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [271]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Self-citationFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725 · see citing works
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · see citing works
NUS-WIDE: a real-world web image database from National University of Singapore
Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., & Zheng, Y. (2009) · Proceedings of the ACM International Conference on Image and Video Retrieval · doi:10.1145/1646396.1646452 · Cited by 3,025 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · see citing works
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works
Self-citationConceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
Self-citationOscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works
Exploring the Limits of Weakly Supervised Pretraining
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., & van der Maaten, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_12 · Cited by 1,224 · see citing works
VSE++: Improving Visual-Semantic Embeddings with Hard NegativesPreprint
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 1,002 · see citing works
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 879 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 788 · see citing works
HashNet: Deep Learning to Hash by Continuation
Cao, Z., Long, M., Wang, J., & Yu, P. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.598 · Cited by 777 · see citing works
Visual Semantic Reasoning for Image-Text Matching
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00475 · Cited by 628 · see citing works
Canonical Correlation Analysis
Thompson, B. (2005) · Encyclopedia of Statistics in Behavioral Science · doi:10.1002/0470013192.bsa068 · Cited by 470 · see citing works
Central Similarity Quantization for Efficient Image and Video Retrieval
Yuan, L., Wang, T., Zhang, X., Tay, F. E., Jie, Z., Liu, W., & Feng, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00315 · Cited by 413 · see citing works
Deep Cauchy Hashing for Hamming Space Retrieval
Cao, Y., Long, M., Liu, B., & Wang, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00134 · Cited by 389 · see citing works
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2020) · arXiv · doi:10.48550/arxiv.2011.10566 · Cited by 381 · see citing works
Understanding image representations by measuring their equivariance and equivalence
Lenc, K., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298701 · Cited by 366 · see citing works
A Decade Survey of Content Based Image Retrieval Using Deep Learning
Dubey, S. R. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3080920 · Cited by 365 · see citing works
Cross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 359 · see citing works
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning
Chen, S., Zhao, Y., Jin, Q., & Wu, Q. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01065 · Cited by 345 · see citing works
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal TransformersPreprint
Huang, Z., Zeng, Z., Liu, B., Fu, D., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00849 · Cited by 344 · see citing works
Learning the Best Pooling Strategy for Visual Semantic Embedding
Chen, J., Hu, H., Wu, H., Jiang, Y., & Wang, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01553 · Cited by 314 · see citing works
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
Li, W., Gao, C., Niu, G., Xiao, X., Liu, H., Liu, J., Wu, H. R., & Wang, H. (2021) · doi:10.18653/v1/2021.acl-long.202 · Cited by 308 · see citing works
CLEAR: Contrastive Learning for Sentence RepresentationPreprint
Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., & Ma, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15466 · Cited by 281 · see citing works
Learning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00608 · Cited by 140 · see citing works
Learning Dual Semantic Relations With Graph Attention for Image-Text Matching
Wen, K., Gu, X., & Cheng, Q. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3030656 · Cited by 131 · see citing works
Self-citationBridging multimedia heterogeneity gap via Graph Representation Learning for cross-modal retrieval
Cheng, Q., & Gu, X. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.11.011 · Cited by 38 · see citing works
Preserving Semantic Neighborhoods for Robust Cross-Modal Retrieval
Thomas, C., & Kovashka, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_19 · Cited by 32 · see citing works
Learning the Best Pooling Strategy for Visual Semantic EmbeddingPreprint
Chen, J., Hu, H., Wu, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.04305 · Cited by 23 · see citing works
Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
Cui, W., Zheng, G., & Wang, W. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.444 · Cited by 17 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
1 of the 56 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search
Bai, Y., Cao, M., Gao, D., Cao, Z., Chen, C., Fan, Z., Nie, L., & Zhang, M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/62 · Cited by 139 · see citing works
Liao, H., Shen, H., Li, Z., Wang, C., Li, G., Bie, Y., & Xu, C. (2024) · Communications in Transportation Research · doi:10.1016/j.commtr.2023.100116 · Cited by 84 · see citing works
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
Lu, H., Fei, N., Huo, Y., Gao, Y., Lu, Z., & Wen, J.-R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01524 · Cited by 76 · see citing works
Yu, J., Liu, J., Cheng, Y., Feng, R., & Zhang, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547868 · Cited by 70 · see citing works
Context-aware Alignment and Mutual Masking for 3D-Language Pre-training
Jin, Z., Hayat, M., Yang, Y., Guo, Y., & Lei, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01057 · Cited by 56 · see citing works
Vision-and-Language Pretrained Models: A Survey
Long, S., Cao, F., Han, S. C., & Yang, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/773 · Cited by 46 · see citing works
Supervised contrastive learning for wafer map pattern classification
Bae, Y., & Kang, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107154 · Cited by 37 · see citing works
LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval
Luo, Z., Zhao, P., Xu, C., Geng, X., Shen, T., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01029 · Cited by 34 · see citing works
Instance-Dictionary Learning for Open-World Object Detection in Autonomous Driving Scenarios
Ma, Z., Zheng, Z., Wei, J., Yang, Y., & Shen, H. T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3322465 · Cited by 33 · see citing works
Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks
Cheng, Q., Wen, K., & Gu, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3217384 · Cited by 29 · see citing works
Self-citationMultiview adaptive attention pooling for image–text retrieval
Ding, Y., Yu, J., Lv, Q., Zhao, H., Dong, J., & Li, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111550 · Cited by 21 · see citing works
Decoupled Cross-Modal Phrase-Attention Network for Image-Sentence Matching
Shi, Z., Zhang, T., Wei, X., Wu, F. H., & Zhang, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3197972 · Cited by 20 · see citing works
Cross-Modal Contrastive Learning for Domain Adaptation in 3D Semantic Segmentation
Xing, B., Ying, X., Wang, R., Yang, J., & Chen, T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25400 · Cited by 20 · see citing works
Geographical Supervision Correction for Remote Sensing Representation Learning
Li, W., Chen, K., & Shi, Z. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3202499 · Cited by 16 · see citing works
Cross-modal alignment and translation for missing modality action recognition
Park, Y., Woo, S., Lee, S., Nugroho, M. A., & Kim, C. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103805 · Cited by 16 · see citing works
ViLEM: Visual-Language Error Modeling for Image-Text Retrieval
Chen, Y., Ma, Z., Zhang, Z., Qi, Z., Yuan, C., Shan, Y., Li, B., Hu, W., Qie, X., & Wu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01060 · Cited by 14 · see citing works
Transformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 68 · see citing works
Cheng, Q., Guo, Q., & Gu, X. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120731 · Cited by 10 · see citing works
Cross-modal contrastive learning for unified placenta analysis using photographs
Pan, Y., Mehta, M., Goldstein, J. A., Ngonzi, J. N. J., Bebell, L. M., Roberts, D. J., Carreon, C. K., Gallagher, K., Walker, R. E., Gernand, A. D., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.101097 · Cited by 9 · see citing works
Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control
Nguyen, T. D.-M., Hendriksen, M., Yates, A., & de Rijke, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_29 · Cited by 9 · see citing works
Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals
Wu, T.-L., Spangher, A., Alipoormolabashi, P., Freedman, M., Weischedel, R., & Peng, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.310 · Cited by 8 · see citing works
How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
Chen, Y., Ma, Z., Zhang, Z., Qi, Z., Yuan, C., Li, B., Pu, J., Shan, Y., Qi, X., & Hu, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02549 · Cited by 7 · see citing works
Gong, Z., Huang, Y., Yu, C., Dai, P., Ge, X., Shen, Y., & Liu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129642 · Cited by 8 · see citing works
Self-citationConditioned Masked Language and Image Modeling for Image-Text Dense Retrieval
Luo, Z., Xi, Y., Zhang, R., Li, G., Zhao, Z., & Ma, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.10 · Cited by 6 · see citing works
Pan, Y., Cai, T., Mehta, M., Gernand, A. D., Goldstein, J. A., Mithal, L. B., Mwinyelle, D., Gallagher, K., & Wang, J. Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43987-2_12 · Cited by 6 · see citing works
Omni-Granularity Embedding Network for Text-to-Image Person Retrieval
Wang, C., Luo, Z., & Li, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688072 · Cited by 3 · see citing works
A dual‐modal graph attention interaction network for person Re‐identification
Wang, W., An, G., & Ruan, Q. (2023) · IET Computer Vision · doi:10.1049/cvi2.12192 · Cited by 3 · see citing works
Fine-Grained Multimodal Alignment for Image-Text Retrieval via Graph Learning
Chen, M., Zhang, X., Qi, L., Li, X., Yang, X., Hoi, S. C. H., Liu, Z., & Yang, M. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02686-y · Cited by 2 · see citing works
Han, H., Lian, C., Xu, B., Zeng, Z., Alhudhaif, A., & Polat, K. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112372 · Cited by 2 · see citing works
Multi-Constraint Relational Semantic Alignment Toward Image-Text Retrieval
Xiao, Y., Zhang, J., & Song, A. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3664964 · Cited by 1 · see citing works
Annotate and retrieve in vivo images using hybrid self-organizing map
Kaur, P., Malhi, A. K., & Pannu, H. S. (2023) · The Visual Computer · doi:10.1007/s00371-023-03126-z · Cited by 1 · see citing works
VEMO: A Versatile Elastic Multi-modal Model for Search-Oriented Multi-task Learning
Fei, N., Jiang, H., Lu, H., Long, J., Dai, Y., Fan, T., Cao, Z., & Lu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_4 · Cited by 1 · see citing works
A Unified Multiple Inducible Co-attentions and Edge Guidance Network for Co-saliency Detection
Tan, Z., & Gu, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15919-0_2 · Cited by 1 · see citing works
Understanding Procedural Knowledge by Sequencing Multimodal Instructional ManualsPreprint
Wu, T.-L., Spangher, A., Alipoormolabashi, P., Freedman, M., Weischedel, R., & Peng, N. (2021) · arXiv (Cornell University)
Trajectory-text retrieval model based on graph neural network
Tang, H., Wang, L., & Luo, Q. (2023) · International Conference on Computer Vision and Data Mining (ICCVDM) · doi:10.1117/12.2660180
MultiMem: Measuring and Mitigating Memorization in Multi-modal Contrastive Learning
Wang, W., Boenisch, F., Michael Backes, A., & Dziedzic, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37029-7_8
- Authors
- Wen, K., Xia, J., Huang, Y., Li, L., Xu, J., & Shao, J.
- Affiliations
- ByteDance (China) (CN)
- Fudan University (CN)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 2188-2197
- OpenAlex
- W3201703290
- Cited by
- 37
- References
- 56
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 8 · 2023: 13 · 2022: 10 · 2021: 1
- FWCI
- 2.62
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- vision-language representation learning, contrastive learning, cross-modal retrieval, text matching, image retrieval, inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
