The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 301–400 of 1,011
Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
Bousselham, W., Petersen, F., Ferrari, V., & Kuehne, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00367 · Cited by 77
“A representative example is OpenSeg [7] that fine-tunes a model using class-agnostic masks and image-text pair data based on ALIGN [9].”
BackgroundMethodClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
Lan, M., Chen, Z., Ke, Y., Wang, X., Feng, L., & Zhang, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72970-6_9 · Cited by 58
Learning Without Forgetting for Vision-Language Models
Zhou, D.-W., Zhang, Y., Wang, Y., Ning, J., Ye, H.-J., Zhan, D., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3540889 · Cited by 102
GraDual: Graph-based Dual-modal Representation for Image-Text Matching
Long, S., Han, S. C., Wan, X., & Poon, J. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00252 · Cited by 57
Iftekhar, A., Chen, H., Kundu, K., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00528 · Cited by 56
Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive Training
Liu, C., Zhang, Y., Wang, H., Chen, W., Wang, F., Huang, Y., Shen, Y.-D., & Wang, L. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3286710 · Cited by 58
Dual-Path Adaptation from Image to Video Transformers
Park, J., Lee, J., & Sohn, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00219 · Cited by 64
Research status and application of artificial intelligence large models in the oil and gas industry
Liu, H., Ren, Y., Li, X., Deng, Y., Wang, Y., Cao, Q., Jinyang, D. U., Lin, Z., & Wang, W. (2024) · Petroleum Exploration and Development · doi:10.1016/s1876-3804(24)60524-0 · Cited by 63
Context-aware Alignment and Mutual Masking for 3D-Language Pre-training
Jin, Z., Hayat, M., Yang, Y., Guo, Y., & Lei, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01057 · Cited by 56
Self-citationFAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks
Xiao, H., Zhu, X., Yu, L., Zhang, L., Song, Y.-Z., & Xiang, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00262 · Cited by 58
Efficient Multimodal Fusion via Interactive Prompting
Li, Y., Quan, R., Zhu, L., & Yang, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00256 · Cited by 60
“Another important line of multimodal fusion work is through pretraining with the large-scale datasets, mostly achieved by self-supervised learning algorithms [1, 11, 24, 30, 36, 37, 45].”
BackgroundMethodSelf-citationScaling Vision-Language Models with Sparse Mixture of Experts
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.758 · Cited by 113
Cross-Modal Adaptive Dual Association for Text-to-Image Person Retrieval
Lin, D., Peng, Y.-X., Meng, J., & Zheng, W. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3355644 · Cited by 55
“CLIP [14] and ALIGN [34] employ contrastive learning loss to learn strong coherent cross-modal representations for cross-modal alignment.”
MethodDomain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation
Du, Z., Li, X., Li, F., Lü, K., Zhu, L., & Li, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02206 · Cited by 57
“Recently, large-scale pre-trained Vision-Language Models (VLMs) have demonstrated impressive successes in various downstream tasks [10, 13, 35, 54].”
BackgroundConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing
Zeng, Z., Zhang, H., Lu, R., Wang, D., Chen, B., & Wang, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02247 · Cited by 56
Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
Radenović, F., Dubey, A., Kadian, A., Mihaylov, T., Vandenhende, S., Patel, Y., Wen, Y., Ramanathan, V., & Mahajan, D. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00673 · Cited by 56
SAN: Side Adapter Network for Open-Vocabulary Semantic Segmentation
Xu, M., Zhang, Z., Wei, F., Hu, H., & Bai, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3311618 · Cited by 54
Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
Maniparambil, M., Vorster, C., Molloy, D., Murphy, N. A., McGuinness, K., & O'Connor, N. E. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00034 · Cited by 59
PØDA: Prompt-driven Zero-shot Domain Adaptation
Fahes, M., Vu, T.-H., Bursuc, A., Pérez, P., & de Charette, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01707 · Cited by 53
CoVR: Learning Composed Video Retrieval from Web Video Captions
Ventura, L., Yang, A., Schmid, C., & Varol, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28334 · Cited by 62
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval
Geigle, G., Pfeiffer, J., Reimers, N., Vulić, I., & Gurevych, I. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00473 · Cited by 60
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Andonian, A., Chen, S., & Hamid, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01594 · Cited by 57
LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models
Bulat, A., & Tzimiropoulos, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02225 · Cited by 64
CT Image Denoising and Deblurring With Deep Learning: Current Status and Perspectives
Lei, Y., Niu, C., Zhang, J., Wang, G., & Shan, H. (2023) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2023.3341903 · Cited by 56
“With transformers’ great performance in various tasks of natural language processing and computer vision, the vision-language large-scale pretrained models have emerged, such as CLIP [175], ALIGN [75], etc.”
A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models
Silva-Rodríguez, J., Hajimiri, S., Ayed, I. B., & Dolz, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02235 · Cited by 64
“Particularly, models such as CLIP [30] and ALIGN [18] have successfully mitigated the distribution discrepancy be-tween text and images, and have shown tremendous zero-shot capabilities on visual recognition tasks, primarily in the context of classification.”
BackgroundMethodFrom screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Transcriptomics-Guided Slide Representation Learning in Computational Pathology
Jaume, G., Oldenburg, L., Vaidya, A., Chen, R. J., Williamson, D. F. K., Peeters, T., Song, A. H., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00920 · Cited by 51
“Representation learning in vision has also evolved towards multimodal vision-language models [1, 36, 48, 49, 51, 67, 74, 82, 94].”
BackgroundOvarNet: Towards Open-Vocabulary Object Attribute Recognition
Chen, K., Jiang, X., Hu, Y. G., Tang, X., Gao, Y. F., Chen, J., & Xie, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02252 · Cited by 55
Towards All-in-One Pre-Training via Maximizing Multi-Modal Mutual Information
Su, W., Zhu, X., Tao, C., Lu, L., Li, B., Huang, G. M., Qiao, Y., Wang, X., Zhou, J., & Dai, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01525 · Cited by 58
Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning
Jiang, Q., Chen, C., Zhao, H., Chen, L., Ping, Q., Tran, S. D., Xu, Y., Zeng, B., & Chilimbi, T. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00740 · Cited by 57
Parameter-Efficient Transfer Learning for Medical Visual Question Answering
Liu, J., Hu, T., Zhang, Y., Yang, F., Hao, J., Lv, J., & Liu, Z. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3311333 · Cited by 52
Detecting Backdoors in Pre-trained Encoders
Feng, S., Tao, G., Cheng, S., Shen, G., Xu, X., Liu, Y., Zhang, K., Ma, S., & Zhang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01569 · Cited by 57
FedSH: Towards Privacy-Preserving Text-Based Person Re-Identification
Ma, W., Wu, X., Zhao, S., Zhou, T., Guo, D., Gu, L., Cai, Z., & Wang, M. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3330091 · Cited by 52
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
Su, Y., Liu, F., Meng, Z., Lan, T., Shu, L., Shareghi, E., & Collier, N. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.191 · Cited by 59
ESA: External Space Attention Aggregation for Image-Text Retrieval
Zhu, H., Zhang, C., Wei, Y., Huang, S., & Zhao, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3253548 · Cited by 49
Li, J., Zhang, Y., Qiang, W., Si, L., Jiao, C., Hu, X., Zheng, C., & Sun, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25216 · Cited by 50
CLIPPO: Image-and-Language Understanding from Pixels Only
Tschannen, M., Mustafa, B., & Houlsby, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01059 · Cited by 65
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Sermanet, P., Ding, T., Zhao, J. W., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610216 · Cited by 108
“Recently many methods [11, 15, 16, 10, 12, 17, 13] have been proposed that aim to train vision-language models (VLMs) on large-scale image-text pair datasets.”
ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation
Lan, M., Chen, Z., Ke, Y., Wang, X., Feng, L., & Zhang, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73113-6_5 · Cited by 67
FlexIT: Towards Flexible Semantic Image Translation
Couairon, G., Grechka, A., Verbeek, J. J., Schwenk, H., & Cord, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01773 · Cited by 49
Teaching Structured Vision & Language Concepts to Vision & Language Models
Doveh, S., Arbelle, A., Harary, S., Schwartz, E., Herzig, R., Giryes, R., Feris, R., Panda, R., Ullman, S., & Karlinsky, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00261 · Cited by 57
MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Ji, Y., Wang, J., Gong, Y., Zhang, L., Zhu, Y., Wang, H., Zhang, J., Sakai, T., & Yang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02228 · Cited by 59
Self-citationEdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment
Shi, C., & Yang, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01441 · Cited by 53
Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
Huang, H., Nie, Z., Wang, Z., & Shang, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29789 · Cited by 55
“Existing models can be classified into three categories based on their architecture: (1) dual-encoder (Radford et al. 2021; Jia et al. 2021); (2) fusion-encoder (Lee et al. 2018; Diao et al. 2021; Chen et al. 2021; Li et al. 2022b; Zhang et al. 2022), and (3) dual-encoder + fusion-encoder (Li et al.…”
BackgroundFashionViL: Fashion-Focused Vision-and-Language Representation Learning
Xiao, H. M., Yu, L., Zhu, X., Zhang, L., Song, Y.-Z., & Xiang, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_37 · Cited by 47
xGQA: Cross-Lingual Visual Question Answering
Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.-M. O., Roth, S., Vulić, I., & Gurevych, I. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.196 · Cited by 67
Vision-and-Language Pretrained Models: A Survey
Long, S., Cao, F., Han, S. C., & Yang, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/773 · Cited by 66
Compression of Generative Pre-trained Language Models via Quantization
Tao, C., Hou, L., Zhang, W., Shang, L., Jiang, X., Liu, Q., Luo, P., & Wong, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.331 · Cited by 52
BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
Oh, C., Hwang, H., Lee, H., Lim, Y., Jung, G., Jung, J., Choi, H., & Song, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02320 · Cited by 50
Cho, Y., Yu, H., & Kang, S. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3340062 · Cited by 56
“Different from the conventional segmentation task [13], [14], [15], [16], [17], [18] based on fixed categories, referring image segmentation aims to find the target object according to the unrestricted language expressions.”
BackgroundUnLoc: A Unified Framework for Video Localization Tasks
Shen, Y., Xiong, X., Nagrani, A., Arnab, A., Wang, Z., Ge, W., Ross, D. A., & Schmid, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01253 · Cited by 60
RLIPv2: Fast Scaling of Relational Language-Image Pre-training
Yuan, H., Zhang, S., Wang, X., Albanie, S., Pan, Y., Feng, T., Jiang, J., Ni, D., Zhang, Y., & Zhao, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01979 · Cited by 53
Mask Grounding for Referring Image Segmentation
Chng, Y. X., Zheng, H. W., Han, Y., Qiu, X., & Huang, G. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02509 · Cited by 62
“3(a), we demonstrate Mask Ground-ing’s superiority over both the standard masked language modeling (MLM) [3, 13, 33, 57] and masked-vision language modeling (MaskedVLM) [37], highlighting our ap-proach’s effectiveness.”
BackgroundArGue: Attribute-Guided Prompt Tuning for Vision-Language Models
Tian, X., Zou, S., Yang, Z., & Zhang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02700 · Cited by 54
“In the evolving field of V&L models [1, 17, 37], crafting text encoder prompts is pivotal for enhancing few-shot performance.”
BackgroundLearning multi-modal representations by watching hundreds of surgical video lectures
Yuan, K., Srivastav, V., Yu, T., Lavanchy, J. L., Marescaux, J., Mascagni, P., Navab, N., & Padoy, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103644 · Cited by 82
All You May Need for VQA are Image Captions
Changpinyo, S., Kukliansy, D., Szpektor, I., Chen, X., Ding, N., & Soricut, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.142 · Cited by 67
Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
Han, K., Liu, Y., Liew, J. H., Ding, H., Liu, J., Wang, Y., Tang, Y., Yang, Y., Feng, J., Zhao, Y., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00080 · Cited by 48
Self-citationDistribution-Aware Prompt Tuning for Vision-Language Models
Cho, E., Kim, J., & Kim, H. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02011 · Cited by 50
Omni-Scan2BIM: A ready-to-use Scan2BIM approach based on vision foundation models for MEP scenes
Wang, B., Chen, Z., Li, M., et al. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105384 · Cited by 45
A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
Yuan, B., & Zhao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3446949 · Cited by 59
“As a rapid-growing hotspot, foundation models such as the vision-language pre-training (VLP) models [145] and the self-supervised pre-training models play an important role in multi-modal research.”
BackgroundFan, J., Yin, Y., Wang, T., Dong, W., Zheng, P., & Wang, L. (2025) · Frontiers of Engineering Management · doi:10.1007/s42524-025-4136-9 · Cited by 53
“Prevalent examples of VLMs include CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which have shown promise in tasks such as image captioning, visual question answering, and multimodal reasoning.”
“BNN - BN = ?”: Training Binary Neural Networks without Batch Normalization
Chen, T., Zhang, Z., Ouyang, X., Liu, Z., Shen, Z., & Wang, Z. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00520 · Cited by 45
Contrastive Data and Learning for Natural Language Processing
Zhang, R., Ji, Y., Zhang, Y., & Passonneau, R. J. (2022) · doi:10.18653/v1/2022.naacl-tutorials.6 · Cited by 47
Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00285 · Cited by 48
Brain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey
Jiao, L., Huang, Z., Lu, X., Liu, X., Yang, Y., Zhao, J., Zhang, J., Hou, B., Yang, S., Liu, F., et al. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3316302 · Cited by 49
Self-citationALIP: Adaptive Language-Image Pre-training with Synthetic Caption
Yang, K., Deng, J., An, X., Li, J., Feng, Z., Guo, J., Yang, J., & Liu, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00273 · Cited by 50
Li, L., Camps, J., Wang, Z. J., et al. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3367409 · Cited by 44
Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval
Han, H., Zheng, Q., Dai, G., Luo, M., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02519 · Cited by 53
“In practice, collecting such ideal data [24] is notoriously labor-intensive and even impossible.”
BackgroundControllable Image Captioning via Prompting
Wang, N., Xie, J., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25360 · Cited by 50
Singha, M., Jha, A., Solanki, B. S., Bose, S., & Banerjee, B. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00196 · Cited by 48
Wang, H., Liu, F., Jiao, L., Wang, J., Hao, Z., Li, S., Li, L., Chen, P., & Liu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28347 · Cited by 53
“Vision language pre-training models like CLIP (Radford et al. 2021) and ALIGN (Jia et al. 2021) are trained using millions of image-text pairs crawled from the Internet without labels.”
BackgroundMethodInfluentialStructure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations
Huang, Y., Tang, J., Chen, Z., Zhang, R., Zhang, X., Chen, W., Zhao, Z., Zhao, Z., Lv, T., Hu, Z., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.28017 · Cited by 57
MURAL: Multimodal, Multitask Representations Across Languages
Jain, A., Guo, M., Srinivasan, K., Chen, T., Kudugunta, S., Jia, C., Yang, Y., & Baldridge, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.293 · Cited by 45
Self-citationDomain-Agnostic Prior for Transfer Semantic Segmentation
Huo, X., Xie, L., Hu, H., Zhou, W., Li, H., & Tian, Q. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00694 · Cited by 44
Vision-Language Pre-Training for Boosting Scene Text Detectors
Song, S., Wan, J., Yang, Z., Tang, J., Cheng, W., Bai, X., & Yao, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01523 · Cited by 42
Hierarchical Prompt Learning for Multi-Task Learning
Liu, Y., Lu, Y., Liu, H., An, Y., Xu, Z., Yao, Z., Zhang, B., Xiong, Z., & Gui, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01048 · Cited by 44
RA-CLIP: Retrieval Augmented Contrastive Language-Image Pre-Training
Xie, C.-W., Sun, S., Xiong, X., Zheng, Y., Zhao, D., & Zhou, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01846 · Cited by 51
Bai, J., Wang, X., Xiao, Z. Q., Zhou, H., Ali, T. A. A., Li, Y., & Jiao, L. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3350927 · Cited by 43
“At present, contrast learning can be performed by single-modal [17], [19], [38], [39], [40], [41], [42] as well as multimodal [43], [44], [45].”
BackgroundKnowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
Kan, B., Wang, T., Lü, W., Zhen, X., Guan, W., & Zheng, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01436 · Cited by 48
Cross-Modal Retrieval With Noisy Correspondence via Consistency Refining and Mining
Ma, X., Yang, M., Li, Y., Hu, P., Lv, J., & Peng, X. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3374221 · Cited by 50
“In multi-modal contrastive learning, pairs are constructed based on the correspondence between cross-modal samples [12], [13], [49], [50].”
BackgroundMini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance
Gao, Z., Chen, Z., Cui, E., Ren, Y., Wang, W., Zhu, J., Tian, H., He, J., He, J., Zhu, X., et al. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00067-6 · Cited by 84
“As a result, these vision encoders are not capable”
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
Yao, Y., Chen, Q., Zhang, A., Ji, W., Liu, Z., Chua, T., & Sun, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.763 · Cited by 53
Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream ImplicationsPreprint
Agarwal, S., Krueger, G. M., Clark, J., Radford, A., Kim, J. W., & Brundage, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02818 · Cited by 161
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., & Li, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01454 · Cited by 56
HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning
Kuo, C.-W., & Kira, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01062 · Cited by 45
Two-stage fine-grained image classification model based on multi-granularity feature fusion
Xu, Y., Wu, S., Yang, M., Yang, M., Wu, Z., Yao, Y., & Wei, Z. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110042 · Cited by 41
CEPrompt: Cross-Modal Emotion-Aware Prompting for Facial Expression Recognition
Zhou, H., Huang, S., Zhang, F., & Xu, C. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3424777 · Cited by 49
“Recently, Vision-Language Pre-training (VLP) models such as CLIP [19], ALIGN [23], LiT [44], CLIPPO [24] are pre-trained with large-scale image-text pairs, demonstrating powerful generalization performance.”
Efficient Sharpness-aware Minimization for Improved Training of Neural NetworksPreprint
Du, J., Yan, H., Feng, J., Zhou, J. T., Zhen, L., Goh, R. S. M., & Tan, V. Y. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03141 · Cited by 84
Concept-Aware Video Captioning: Describing Videos With Effective Prior Information
Yang, B., Cao, M., & Zou, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3307969 · Cited by 39
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks
Cai, Z., Kwon, G., Ravichandran, A., Bas, E., Tu, Z., Bhotika, R., & Soatto, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_17 · Cited by 48
Neuron-Based Spiking Transmission and Reasoning Network for Robust Image-Text Retrieval
Li, W., Ma, Z., Deng, L.-J., Xiaopeng Fan, B., & Tian, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3233042 · Cited by 40
Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval
Kim, J. M., Koepke, A. S., Schmid, C., & Akata, Z. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00257 · Cited by 41
Miao, J., Zhou, S.-P., Zhou, G., Wang, K.-N., Yang, M., Zhou, S.-J., & Chen, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3336534 · Cited by 41
“With the increase of the scale of data and model, training a foundation model to tackle a wide range of tasks has attracted great interest in the research community, such as ChatGPT, CLIP [72], and ALIGN [73].”
BackgroundSelf-citationHuang, Z., Zhou, A., Lin, Z., Cai, M., Wang, H., & Lee, Y. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01073 · Cited by 45
Implicit Temporal Modeling with Learnable Alignment for Video Recognition
Tu, S., Dai, Q., Wu, Z., Cheng, Z.-Q., Hu, H., & Jiang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01825 · Cited by 52
Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network
Han, C., Zhong, Y., Li, D., Han, K., & Ma, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00106 · Cited by 40
Perceptual Grouping in Contrastive Vision-Language Models
Ranasinghe, K., McKinzie, B., Ravi, S., Yang, Y., Toshev, A., & Shlens, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00513 · Cited by 57
Self-citationMasQCLIP for Open-Vocabulary Universal Image Segmentation
Xu, X., Xiong, T., Ding, Z., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00088 · Cited by 40
FIRE: Food Image to REcipe generation
Chhikara, P., Chaurasia, D., Jiang, Y., Masur, O., & Ilievski, F. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00800 · Cited by 50
StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization
Bose, S., Jha, A., Fini, E., Singha, M., Ricci, E., & Banerjee, B. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00545 · Cited by 47
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
