The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 501–600 of 1,011
Wang, M., Su, L., Yan, C., Xu, S. H., Yuan, P., Jiang, X., & Zhang, B. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3439395 · Cited by 38
“Owing to their self-supervised or semi-supervised training on voluminous data, these models are capable of quickly adapting to a diverse range of downstream tasks, demonstrating remarkable zero-shot inference capabilities [18], [19], [43].”
BackgroundMV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
Jin, X., Zhang, B., Gong, W., Xu, K., Deng, X., Wang, P., Zhang, Z., Shen, X., & Feng, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02563 · Cited by 58
Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
Barsellotti, L., Amoroso, R., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00354 · Cited by 38
“Building upon the success of large-scale vision-language models in zero-shot classification [16, 34], previous works on open-vocabulary segmentation have investigated strategies to transfer the multimodal image-text alignment toward finer granularity ( i.e ., region or pixel level) [10, 13, 22, 43,…”
BackgroundOpen-Vocabulary Semantic Segmentation with Image Embedding Balancing
Shan, X., Wu, D., Zhu, G., Shao, Y., Sang, N., & Gao, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02684 · Cited by 31
“EN-B7 [32] is the image backbone used by ALIGN [12].”
MethodPLOVAD: Prompting Vision-Language Models for Open Vocabulary Video Anomaly Detection
Xu, C., Xu, K., Jiang, X., & Sun, T. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3528108 · Cited by 33
“Additionally, image-based vision-language models (I-VLMs), exemplified by CLIP [32] and ALIGN [33], pre-train themselves on large-scale image-text pairs, demonstrate remarkable zero-shot performance on various vision tasks [34].”
M³amba: CLIP-Driven Mamba Model for Multi-Modal Remote Sensing Classification
Cao, M., Xie, W., Zhang, X., Zhang, J., Jiang, K., Lei, J., & Li, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3549750 · Cited by 31
“Vision-language models (VLMs), such as CLIP [7] and ALIGN [19], are trained on billion-scale, often noisy image-text datasets.”
Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts
Tian, Y., & Ha, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-03789-4_18 · Cited by 42
Open-vocabulary Attribute Detection
Bravo, M. A., Mittal, S., Ging, S., & Brox, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00680 · Cited by 36
Zhu, H., Wei, Y., Liang, X., Zhang, C., & Zhao, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02034 · Cited by 32
Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
Li, H., Zhu, J., Jiang, X., Zhu, X., Li, H., Yuan, C., Wang, X., Qiao, Y., Wang, X., Wang, W., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00264 · Cited by 41
Instruction-ViT: Multi-modal prompts for instruction learning in vision transformer
Xiao, Z., Chen, Y., Yao, J., Zhang, L., Liu, Z., Wu, Z., Yu, X., Pan, Y., Zhao, L., Ma, C., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102204 · Cited by 51
Self-citationA Dual-Feature-Based Adaptive Shared Transformer Network for Image Captioning
Shi, Y., Xia, J., Zhou, M., & Cao, Z. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3353830 · Cited by 31
Learning Semantic Polymorphic Mapping for Text-Based Person Retrieval
Li, J., Jiang, M., Kong, J., Tao, X., & Luo, X. L. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3410129 · Cited by 31
Visual and Language Collaborative Learning for RGBT Object Tracking
Wang, J., Liu, F., Jiao, L., Gao, Y., Wang, H., Li, S., Li, L., Chen, P., & Liu, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3436878 · Cited by 30
“Some models [35], [36], [37] have adjusted the similarity measurement between text and images based on CLIP.”
Explicit High-Level Semantic Network for Domain Generalization in Hyperspectral Image Classification
Wang, X., Dong, S., Zheng, X., Lu, R., & Jia, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3495765 · Cited by 30
Zhan, C., Peng, P., Wang, H., Wang, G., Lin, Y., Chen, T., & Wang, H. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103464 · Cited by 32
Fine-Grained Visual–Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Long, Y., Han, J., Huang, R., Xu, H., Zhu, Y., Xu, C., & Liang, X. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3293484 · Cited by 29
Yang, S.-K., Zhang, X., Chen, Y.-F., Jiang, Y., Feng, Q., Pu, L., & Sun, F. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110868 · Cited by 30
Self-citationSceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text
Chowdhury, P. N., Bhunia, A. K., Sain, A., Koley, S., Xiang, T., & Song, Y.-Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01056 · Cited by 31
Joint representation learning for text and 3D point cloud
Huang, R., Pan, X., Zheng, H., Jiang, H., Xie, Z., Wu, C., Song, S., & Huang, G. M. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110086 · Cited by 29
Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 33
Self-citationSemantic-Enhanced Image Clustering
Cai, S., Qiu, L., Chen, X., Zhang, Q., & Chen, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25841 · Cited by 28
Mumuni, F., & Mumuni, A. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101188 · Cited by 34
“The zero-and few-shot generalization property of CLIP which has been confirmed by several studies (e.g., [250], [251]) is critical in the open-world neural decoding task.”
BackgroundMulti-Modal Attribute Prompting for Vision-Language Models
Liu, X., Wu, J., Yang, W., Zhou, X., & Zhang, T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3424566 · Cited by 32
“Large-scale pre-trained Vision-Language Models (VLMs), such as CLIP [38] and ALIGN [17], have demonstrated promising generalization power and transfer-ability on a wide range of downstream tasks, including image classification [38], object detection [2,20,24] and 3D understanding [36,42,51].”
BackgroundLAMM: Label Alignment for Multi-Modal Prompt Learning
Gao, J., Ruan, J., Xiang, S., Yu, Z., Ji, K. F., Xie, M., Liu, T., & Fu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27950 · Cited by 30
Exploration of Class Center for Fine-Grained Visual Classification
Yao, H., Miao, Q., Zhao, P., Li, C., Li, X., Feng, G., & Liu, R. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3406443 · Cited by 32
“Although ALIGN outperforms our method on CAR dataset, ALIGN is pretrained on a large-scale noisy image-text dataset (LSNITD) [63], which includes 1.8B image-text pairs.”
MethodTransductive Zero-Shot and Few-Shot CLIP
Martin, S., Huang, Y., Shakeri, F., Pesquet, J.-C., & Ayed, I. B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02722 · Cited by 29
“The emergence of large-scale vision-language models like CLIP [40] has marked a significant turning point in representation learning [24, 29, 54].”
BackgroundA survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 45
Zanella, M., & Ayed, I. B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02245 · Cited by 38
“Vision-language models, pretrained on vast sets of image-text pairs, have emerged as powerful tools for learning cross-modal representations [25, 30, 46, 60, 62, 63].”
BackgroundMittal, H., Agarwal, N., Lo, S.-Y., & Lee, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01758 · Cited by 33
“Some models [20, 29, 35, 48, 62] merge visual and linguistic modalities by co-training text and image encoders using contrastive loss on large datasets containing image-caption pairs.”
Tokenize Anything via Prompting
Pan, T., Tang, L., Wang, X., & Shan, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72970-6_19 · Cited by 40
Resistive memory-based zero-shot liquid state machine for multimodal event data learning
Lin, N., Wang, S., Li, Y., Wang, B., Shi, S., He, Y., Zhang, W., Yu, Y., Zhang, Y., Zhang, X., et al. (2025) · Nature Computational Science · doi:10.1038/s43588-024-00751-z · Cited by 32
Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Gur, S., Neverova, N., Stauffer, C., Lim, S.-N., Kiela, D., & Reiter, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.11 · Cited by 29
Breaking with Fixed Set Pathology Recognition Through Report-Guided Contrastive Training
Seibold, C., Reiß, S., Sarfraz, M. S., Stiefelhagen, R., & Kleesiek, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_66 · Cited by 30
ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
Zhang, X., Sha, Y., Kampffmeyer, M., Xie, Z., Jie, Z., Huang, C., Peng, J., & Liang, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548230 · Cited by 28
Iterative Proposal Refinement for Weakly-Supervised Video Grounding
Cao, M., Wei, F., Xu, C., Geng, X., Chen, L., Zhang, C., Zou, Y., Shen, T., & Jiang, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00631 · Cited by 28
Low-Resolution Object Recognition With Cross-Resolution Relational Contrastive Distillation
Zhang, K., Ge, S., Shi, R., & Zeng, D. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3310042 · Cited by 29
“W ITH the rapid development of deep learning, deep models have demonstrated remarkable success in various visual recognition applications [1], [2], [3], [4].”
BackgroundInfluentialUnsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving
Najibi, M., Ji, J., Zhou, Y., Qi, C., Yan, X.-C., Ettinger, S., & Anguelov, D. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.00790 · Cited by 54
Multistain Pretraining for Slide Representation Learning in Pathology
Jaume, G., Vaidya, A., Zhang, A., H. Song, A., J. Chen, R., Sahai, S., Mo, D., Madrigal, E., Phi Le, L., & Mahmood, F. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73414-4_2 · Cited by 27
CgT-GAN: CLIP-guided Text GAN for Image Captioning
Yu, J., Li, H., Hao, Y., Zhu, B., Xu, T., & He, X. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611891 · Cited by 28
The State of the Art for Cross-Modal Retrieval: A Survey
Zhou, K., Hassan, F. H., & Gan, K. H. (2023) · IEEE Access · doi:10.1109/access.2023.3338548 · Cited by 29
Wu, M., Huang, Q., Gao, S., & Zhang, Z. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103591 · Cited by 33
Exploring the Application of Large-Scale Pre-Trained Models on Adverse Weather Removal
Tan, Z., Wu, Y., Liu, Q., Chu, Q., Lü, L., Ye, J., & Yu, N. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3368961 · Cited by 33
GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learning
Xu, G., Chai, J. Y., & Kordjamshidi, P. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00567 · Cited by 30
“Large VMLs are pre-trained to learn the semantic alignment between vision and language modalities in different levels [7,23].”
BackgroundMCPL: Multi-Modal Collaborative Prompt Learning for Medical Vision-Language Model
Wang, P., Zhang, H., & Yuan, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3418408 · Cited by 28
“As shown in Table V, we perform our pipeline on three well-established V-L models ALIGN [3], FLAVA [23], and CLIP [63], and then conduct downstream tasks on MIMIC-CXR and Object-CXR to verify their performance after prompt tuning.”
Exploring Regional Clues in CLIP for Zero-Shot Semantic Segmentation
Zhang, Y., Guo, M.-H., Wang, M., & Hu, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00315 · Cited by 27
“Vision-language models, like those in [19, 34, 49], are designed to understand the complex connections between visual elements and their textual explanations.”
BackgroundSelf-Supervised Facial Representation Learning with Facial Region Awareness
Gao, Z., & Patras, I. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00203 · Cited by 29
“Further works perform visual-language pre-training by applying contrastive objective to image-text pairs [29, 35, 47].”
BackgroundID-like Prompt Learning for Few-Shot Out-of-Distribution Detection
Bai, Y., Han, Z., Cao, B., Jiang, X., Hu, Q., & Zhang, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01655 · Cited by 37
CapsFusion: Rethinking Image-Text Data at Scale
Yu, Q., Sun, Q., Zhang, X., Cui, Y., Zhang, F., Cao, Y. Q., Wang, X., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01330 · Cited by 58
“[47], where real-world information is substituted with generic concepts . and sub-optimal for model training [27, 34].”
BackgroundCross-Modal Remote Sensing Image–Text Retrieval via Context and Uncertainty-Aware Prompt
Wang, Y., Tang, X., Ma, J., Zhang, X., Liu, F., & Jiao, L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3458898 · Cited by 34
“Nonfused-based methods [26], [45] process images and texts separately using two encoders.”
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
Wei, C., Chen, Y., Chen, H., Hu, H., Zhang, G., Fu, J., Ritter, A., & Chen, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73021-4_23 · Cited by 95
Self-citationPrompt-Based Concept Learning for Few-Shot Class-Incremental Learning
Li, S., Liu, F., Jiao, L., Li, L., Chen, P., Liu, X., & Ma, W. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3525545 · Cited by 27
“Representative models such as ALIGN [65] and CLIP [31] use large-scale image-text pairing for contrastive learning, demonstrating their excellent performance in zero-shot [66] and few-shot learning [67].”
Liu, R.-Y., Liu, Y., Wu, M., Xin, W., Miao, Q.-G., Liu, X.-Z., & Li, L. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111377 · Cited by 36
DVIS++: Improved Decoupled Framework for Universal Video Segmentation
Zhang, T., Tian, X., Zhou, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3552694 · Cited by 27
Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
Wang, Y., Dong, J., Liang, T., Zhang, M., Cai, R., & Wang, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548003 · Cited by 32
Delving into Shape-aware Zero-shot Semantic Segmentation
Liu, X., Tian, B., Wang, Z., Wang, R., Sheng, K., Zhang, B., Zhao, H., & Zhou, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00293 · Cited by 27
Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities
Hu, H., Luan, Y., Chen, Y., Khandelwal, U., Joshi, M., Lee, K., Toutanova, K., & Chang, M.-W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01108 · Cited by 46
Self-citationSimple Image-Level Classification Improves Open-Vocabulary Object Detection
Fang, R., Pang, G., & Bai, X. P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27939 · Cited by 27
You Never Cluster AlonePreprint
Shen, Y., Shen, Z., Wang, M., Qin, J., Torr, P. H. S., & Shao, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01908 · Cited by 29
UFO: A UniFied TransfOrmer for Vision-Language Representation LearningPreprint
Wang, J., Hu, X., Gan, Z., Yang, Z., Dai, X., Liu, Z., Lu, Y., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10023 · Cited by 50
Zero-Shot Generative Model Adaptation via Image-Specific Prompt Learning
Guo, J., Wang, C., Wu, Y., Zhang, E., Wang, K., Xu, X., Song, S., Shi, H., & Huang, G. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01106 · Cited by 26
Lai, Z., Li, Z., Oliveira, L. C., Chauhan, J., Dugger, B. N., & Chuah, C. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00251 · Cited by 27
Self-citationVicTR: Video-conditioned Text Representations for Activity Recognition
Kahatapitiya, K., Arnab, A., Nagran, A., & Ryoo, M. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01755 · Cited by 26
A Simple Recipe for Language-Guided Domain Generalized Segmentation
Fahes, M., Vu, T.-H., Bursuc, A., Pérez, P., & de Charette, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02211 · Cited by 31
“Lately, contrastive language image pretraining (CLIP) [24, 43, 59, 60] has demonstrated that transferable visual representations could be learned from the sole supervision of loose natural language descriptions at very large scale.”
BackgroundCLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow
Pan, C., Yaman, B., Velipasalar, S., & Ren, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01441 · Cited by 26
“Vision-language models (VLMs) have shown great promise for learning good representations for variety of downstream tasks [1, 14, 23, 24, 35, 37].”
BackgroundDreamLIP: Language-Image Pre-training with Long Captions
Zheng, K., Zhang, Y., Wu, W., Lu, F., Ma, S., Jin, X., Chen, W., & Shen, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72649-1_5 · Cited by 46
Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning
Li, Y., Guo, W., Yang, X., Liao, N., He, D., Zhou, J., & Yu, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73016-0_25 · Cited by 36
DOCCI: Descriptions of Connected and Contrasting Images
Onoe, Y., Rane, S., Berger, Z., Bitton, Y., Cho, J., Garg, R., Ku, A., Parekh, Z., Pont-Tuset, J., Tanzer, G., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73027-6_17 · Cited by 69
Self-citationExploring Embodied Multimodal Large Models: Development, datasets, and future directions
Chen, S., Wu, Z., Zhang, K., Li, C., Zhang, B., Ma, F., Yu, F. R., & Li, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103198 · Cited by 39
Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation
Cheng, N., Xu, J., Guan, C., Gao, J., Wang, W., Li, Y., Meng, F., Zhou, J., Fang, B., & Han, W. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103305 · Cited by 49
Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval
Changpinyo, S., Pont-Tuset, J., Ferrari, V., & Soricut, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01192 · Cited by 26
Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation
Cheng, R., Wu, B., Zhang, P., Vajda, P., & Gonzalez, J. E. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00348 · Cited by 26
Achieving Human Parity on Visual Question Answering
Yan, M., Xu, H., Li, C., Tian, J., Bi, B., Wang, W., Xu, X., Zhang, J., Huang, S., Huang, F., et al. (2022) · ACM Transactions on Information Systems · doi:10.1145/3572833 · Cited by 28
NLIP: Noise-Robust Language-Image Pre-training
Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C., & Liang, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25172 · Cited by 30
Fine-Grained Retrieval Prompt Tuning
Wang, S., Chang, J., Wang, Z., Li, H., Ouyang, W., & Tian, Q. C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25363 · Cited by 27
Distilled Dual-Encoder Model for Vision-Language Understanding
Wang, Z., Wang, W., Zhu, H., Liu, M., Qin, B., & Wei, F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.608 · Cited by 34
UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning
Zou, H., Shen, M., Chen, C., Hu, Y., Rajan, D., & Chng, E. S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.41 · Cited by 28
Ju, C., Zheng, K., Liu, J., Zhao, P., Zhang, Y., Chang, J., Tian, Q. C., & Wang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01417 · Cited by 27
Test of Time: Instilling Video-Language Models with a Sense of Time
Bagad, P., Tapaswi, M., & Snoek, C. G. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00247 · Cited by 31
Stare at What You See: Masked Image Modeling without Reconstruction
Xue, H., Gao, P., Li, H., Qiao, Y., Sun, H., Li, H., & Luo, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02177 · Cited by 32
Room-Object Entity Prompting and Reasoning for Embodied Referring Expression
Gao, C., Liu, S., Chen, J., Wang, L., Wu, Q., Li, B. H., & Tian, Q. C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3326851 · Cited by 30
PIDRo: Parallel Isomeric Attention with Dynamic Routing for Text-Video Retrieval
Guan, P., Pei, R., Shao, B., Liu, J., Li, W., Gu, J., Xu, H., Xu, S., Yan, Y., & Lam, E. Y. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01025 · Cited by 25
Zero-Shot Video Grounding With Pseudo Query Lookup and Verification
Lu, Y., Quan, R., Zhu, L., & Yang, Y. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3365249 · Cited by 26
“Models such as CLIP [39] and ALIGN [20] have shown impressive results in downstream tasks.”
MethodSelf-citationFederated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning
Su, S., Yang, M., Li, B., & Xue, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29434 · Cited by 30
Learning Vision from Models Rivals Learning Vision from Data
Tian, Y., Fan, L., Chen, K., Katabi, D., Krishnan, D., & Isola, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01504 · Cited by 53
“An alternative paradigm learns visual representations from text supervision [49, 71], e.g ., CLIP [71].”
BackgroundFrequency-Aware Multi-Modal Fine-Tuning for Few-Shot Open-Set Remote Sensing Scene Classification
Zhang, J., Rao, Y., Huang, X., Li, G., Zhou, X., & Zeng, D. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3372416 · Cited by 28
CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free
Wysoczańska, M., Ramamonjisoa, M., Trzcinski, T. P., & Siméoni, O. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00143 · Cited by 25
Motion-Aware Mask Feature Reconstruction for Skeleton-Based Action Recognition
Zhu, X., Shu, X., & Tang, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3416732 · Cited by 26
“However, models like CLIP [25] and ALIGN [26], which utilize image-text pre-training, have revealed the vast potential of employing natural language supervision for scalable visual pre-training.”
Retrieval-Augmented Open-Vocabulary Object Detection
Kim, J., Cho, E., Kim, S., & Kim, H. J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01650 · Cited by 29
“For instance, CLIP [19] and ALIGN [12] are widely used in OVD to classify unseen objects [8,16,28].”
MethodAAPL: Adding Attributes to Prompt Learning for Vision-Language Models
Kim, G., Kim, S., & Lee, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00164 · Cited by 27
“Prominent models such as CLIP [39] and ALIGN [17], which have advanced through large-scale web data utilization, employ self-supervised learning for enhanced textual and visual alignment.”
MethodVisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions
Moon, S., Woo, H., Park, H., Jung, H., Mahjourian, R., Chi, H., Lim, H., Kim, S., & Kim, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72658-3_21 · Cited by 39
Few-Shot Adaptation of Medical Vision-Language Models
Shakeri, F., Huang, Y., Silva-Rodríguez, J., Bahig, H., Tang, A., Dolz, J., & Ayed, I. B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_52 · Cited by 26
Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation
Jiao, S., Zhu, H., Huang, J., Zhao, Y., Wei, Y., & Shi, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_23 · Cited by 30
ScaleDet: A Scalable Multi-Dataset Object Detector
Chen, Y., Wang, M., Mittal, A., Xu, Z., Favaro, P., Tighe, J., & Modolo, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00704 · Cited by 26
Self-citationLearning to Detect and Segment for Open Vocabulary Object Detection
Wang, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00681 · Cited by 26
Open-vocabulary Panoptic Segmentation with Embedding Modulation
Chen, X., Li, S., Lim, S.-N., Torralba, A. B., & Zhao, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00111 · Cited by 32
Contrastive Feature Masking Open-Vocabulary Vision Transformer
Kim, D., Angelova, A., & Kuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01430 · Cited by 25
Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search
Liang, M., Du, J., Liang, Z., Xing, Y., Huang, W., & Xue, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29280 · Cited by 26
“CLIP (Rad-ford et al. 2021) and ALIGN (Jia et al. 2021) are the recent proposed multi-modal pre-training models based on contrastive learning, which form a task-agnostic model by predicting which text matches which image.”
BackgroundOpen-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Sarch, G., Wu, Y., Tarr, M., & Fragkiadaki, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.226 · Cited by 25
Wu, F., Gao, B., Pan, X., Li, L., Ma, Y., Liu, S., & Liu, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103772 · Cited by 25
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
