The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 601–700 of 1,011
IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-Training
Liu, C., Cheng, S. L., Shi, M., Shah, A., Bai, W., & Arcucci, R. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3449690 · Cited by 30
“Recent methods such as CLIP [2], ALIGN [13], Florence [14], LiT [15] and ALBEF [16] have shown significant progress in improving the representation learning for natural images and languages, although they require substantial training data and computational resources [2], [17], [18].”
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval
Koley, S., Bhunia, A. K., Sain, A., Chowdhury, P. N., Xiang, T., & Song, Y.-Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01562 · Cited by 26
“Thanks to internet-scale paired image-text datasets TBIR has become highly competitive and one of the most active areas of research [51], leading to expansive techniques like Oscar [36], CLIP [51], ALIGN [27], etc.”
MethodPhan, V. M. H., Xie, Y., Qi, Y., Liu, L., Liu, L., Zhang, B., Liao, Z., Wu, Q., To, M., & Verjans, J. W. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01092 · Cited by 26
“Sub-sequently, we present our multi-aspect vision-language matching (MAVL) framework, which decomposes disease entities into a set of visual aspects leveraging the knowledge of medical experts and LLMs (Sec.”
Soft Prompt Generation for Domain Generalization
Bai, S., Zhang, Y., Zhou, W., Luan, Z., & Chen, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72646-0_25 · Cited by 25
Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI
Lee, G.-G., & Zhaı, X. (2025) · TechTrends · doi:10.1007/s11528-024-01035-z · Cited by 27
“For example, in contrastive learning algorithms, which have been typically used to train milestone VLMs such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), textual and visual data pairs are converted through encoders and mapped to each other.”
Facing & mitigating common challenges when working with real-world data: The Data Learning Paradigm
Lever, J., Cheng, S. L., Quilodrán-Casas, C., Liu, C., Fan, H., Platt, R. W., Rakotoharisoa, A., Johnson, E., Li, S., Shang, Z., et al. (2025) · Journal of Computational Science · doi:10.1016/j.jocs.2024.102523 · Cited by 29
Cui, C., Liu, Z., Gong, S., Zhu, L., Zhang, C., & Liu, H. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3541868 · Cited by 25
“Contrastive VLMs, such as CLIP [6] and ALIGN [28], have emerged as the predominant framework in this field.”
CMA-CLIP: Cross-Modality Attention Clip for Text-Image Classification
Fu, J., Xu, S., Liu, H., Liu, Y., Xie, N., Wang, C.-C., Liu, J., Sun, Y., & Wang, B. K. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897323 · Cited by 28
Contrastive Vision-Language Pre-training with Limited Resources
Cui, Q., Zhou, B., Guo, Y. T., Yin, W., Wu, H., Yoshie, O., & Chen, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_14 · Cited by 29
Self-citationLEMaRT: Label-Efficient Masked Region Transform for Image Harmonization
Liu, S., Huynh, C. P., Chen, C., Arap, M., & Hamid, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01754 · Cited by 25
Blending-NeRF: Text-Driven Localized Editing in Neural Radiance Fields
Song, H., Choi, S., Do, H., Lee, C., & Kim, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01323 · Cited by 24
Concept-Guided Prompt Learning for Generalization in Vision-Language Models
Zhang, Y. H., Zhang, C., Yu, K., Tang, Y., & He, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28568 · Cited by 30
Cross‐Modal Graph Contrastive Learning with Cellular Images
Zheng, S., Rao, J., Zhang, J., Zhou, L., Xie, J., Cohen, E., Lu, W., Li, C., & Yang, Y. (2024) · Advanced Science · doi:10.1002/advs.202404845 · Cited by 23
Self-citationTowards Open-World Text-Guided Face Image Generation and ManipulationPreprint
Xia, W., Yang, Y., Xue, J., & Wu, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08910 · Cited by 39
Self-citationMasking-Based Cross-Modal Remote Sensing Image–Text Retrieval via Dynamic Contrastive Learning
Zhao, Z., Miao, X., He, C., Hu, J., Min, B., Gao, Y., Liu, Y., & Pharksuwan, K. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3406897 · Cited by 24
Dai, K., Shao, J., Gong, B., Jing, L., & Chen, Y. (2024) · Aquacultural Engineering · doi:10.1016/j.aquaeng.2024.102460 · Cited by 24
Self-citationLarge Language Models are Good Prompt Learners for Low-Shot Image Classification
Zheng, Z., Wei, J., Hu, X., Zhu, H., & Nevatia, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02688 · Cited by 26
“Vision-Language (VL) models trained on large-scale web data, such as CLIP [32] and ALIGN [14], provide a new paradigm due to their generalization capabilities that includes zero-shot classification, and have been used in recent work [17–19, 23, 24, 48, 49].”
MethodZhao, Y., Zhang, M., Yang, B., Zhang, Z., Kang, J., & Gong, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.02.009 · Cited by 26
VLDeformer: Vision–Language Decomposed Transformer for fast cross-modal retrieval
Zhang, L., Wu, H., Chen, Q., Deng, Y., Siebert, J. I., Li, Z., Han, Y., Kong, D., & Cao, Z. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109316 · Cited by 25
Self-citationQuery-Adaptive Late Fusion for Hierarchical Fine-Grained Video-Text Retrieval
Ma, W., Chen, Q., Liu, F., Zhou, T., & Cai, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3214208 · Cited by 23
Towards Fast and Accurate Image-Text Retrieval With Self-Supervised Fine-Grained Alignment
Zhuang, J., Yu, J., Ding, Y., Qu, X., & Hu, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3280734 · Cited by 25
Probabilistic Prompt Learning for Dense Prediction
Kwon, H., Song, T., Jeong, S., Kim, J., Jang, J., & Sohn, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00654 · Cited by 26
Non-Contrastive Learning Meets Language-Image Pre-Training
Zhou, J., Li, D., Gan, Z., Wang, L., & Wei, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01061 · Cited by 30
Park, S. J., Lee, E. S., Shin, K. S., Lee, J. E., & Ye, J. C. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103021 · Cited by 27
APoLLo : Unified Adapter and Prompt Learning for Vision Language Models
Chowdhury, S., Nag, S., & Manocha, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.629 · Cited by 31
“…approaches pursuing object detection (Bangalath et al., 2022; Zang et al., 2022; Zhou et al., 2022c) and, few-shot image recognition (Zhang et al., 2021; Gao et al., 2021b; Kim et al., 2021) that fare much better compared to off-the-shelf VLP models (Radford et al., 2021; Jia et al., 2021).”
BackgroundMethodRetrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Pan, J., Lin, Z., Ge, Y., Zhu, X., Zhang, R., Wang, Y., Qiao, Y., & Li, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00035 · Cited by 35
Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data
Wu, Z., Weng, Z., Peng, W., Yang, X., Li, A., Davis, L. S., & Jiang, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3357503 · Cited by 27
“Recent interest has shifted towards pretraining models on large-scale visiontext datasets, demonstrating remarkable success in zero-shot image classification [4, 24].”
BackgroundZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection
Phan, T., Vo-Ho, V.-K., Le, D., Doretto, G., Adjeroh, D., & Le, N. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00689 · Cited by 25
“The recent achievements on vision-language (ViL) pre-trained models with representatives such as CLIP [40], ALIGN [16], UniCL [52], Grounding DINO [30], have not only been beneficial to ZS image understanding tasks [5,11,12,29,36,39,56,60] but promoted the generalizability of video analysis issues…”
BackgroundMethodJoint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning
Kong, Y., Cheng, Y., Chen, Y., & Wang, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3417475 · Cited by 23
Self-citationChoi, S. H., Kim, M., & Lee, J. Y. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102836 · Cited by 24
ProS: Prompting-to-Simulate Generalized Knowledge for Universal Cross-Domain Retrieval
Fang, K., Song, J., Gao, L., Zeng, P., Cheng, Z.-Q., Li, X., & Shen, H. T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01637 · Cited by 26
“Subsequently, ALIGN [16] scales the training dataset to billions and gains a better vision-language representation.”
BackgroundVeCLIP: Improving CLIP Training via Visual-Enriched Captions
Lai, Z., Zhang, H., Zhang, B., Wu, W., Bai, H., Timofeev, A., Du, X., Gan, Z., Shan, J., Chuah, C., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_7 · Cited by 64
“To show the data efficiency of VeCLIP, we include ALIGN [17], pre-trained on 1.8B data (denoted as 1.8B-ALIGN), and our in-house CLIP [32] model trained on 1.4B data (denoted as 1.4B-CLIP) as baselines trained at a significantly larger scale.”
BackgroundMethodSelf-citationText-Driven Traffic Anomaly Detection With Temporal High-Frequency Modeling in Driving Videos
Liang, R., Li, Y., Zhou, J., et al. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3390173 · Cited by 22
“Among them, contrastive language-image pre-training methods have achieved remarkable results in many computer vision tasks such as image classification [13], [14], object detection [16], [40], semantic segmentation [17], [41] and image retrieval [42], [43].”
BackgroundMethodSemantic-aware frame-event fusion based pattern recognition via large vision–language models
Li, D., Jin, J., Zhang, Y., Zhong, Y., Wu, Y., Chen, L., Wang, X., & Luo, B. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111080 · Cited by 23
Language-Inspired Relation Transfer for Few-Shot Class-Incremental Learning
Zhao, Y., Li, J., Song, Z., & Tian, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3492328 · Cited by 22
Li, J., Zhang, Y., Zeng, Y., Ye, C., Xu, W. Z., Ben, X., Wang, F., & Zhang, J. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3526815 · Cited by 24
“Numerous studies have already demonstrated the efficacy of such a dataset for other vision recognition tasks [144], [145], [146].”
MA-FSAR: Multimodal Adaptation of CLIP for few-shot action recognition
Xing, J., Zhao, J., Xu, C., Wang, M., Dai, G., Liu, Y., Wang, J., & Li, X. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111902 · Cited by 24
Domain-Controlled Prompt Learning
Cao, Q., Xu, Z., Chen, Y., Ma, C., & Yang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27853 · Cited by 29
Self-citationEmpowering Unsupervised Domain Adaptation with Large-scale Pre-trained Vision-Language Models
Lai, Z., Bai, H., Zhang, H., Du, X., Shan, J., Yang, Y., Chuah, C., & Cao, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00267 · Cited by 27
“Hence these works [52, 60, 61] use ViT as their encoder and achieve superior results than CNNs. Vision-Language Models (VLMs) have shown promising results in learning generic visual representations [24, 39, 44, 68] with language-vision alignment.”
BackgroundSelf-citationZero-Shot Video Moment Retrieval from Frozen Vision-Language Models
Luo, D., Huang, J., Gong, S., Jin, H., & Liu, Y. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00538 · Cited by 23
Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 56
Self-citationBA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model
Song, Y., Zhou, Q., Li, X., Fan, D.-P., Lu, X., & Ma, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00305 · Cited by 26
“Recently, the computer vision community [9, 22, 26, 29– 31, 49, 50, 58, 66, 74, 83–90] has experienced a surge in the development of various foundation models [21, 35, 54].”
BackgroundLin, H., Bai, H., Liu, Z., Hou, L., Sun, M., Song, L., Wei, Y., & Surr, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02584 · Cited by 25
“Benefiting from the efficiency of contrastive learning, vision-language pre-trained models like [10, 14, 18, 27, 42, 61, 64, 65, 71] have achieved advanced capability across downstream tasks.”
SILC: Improving Vision Language Pretraining with Self-distillation
Naeem, M. F., Xian, Y., Zhai, X., Hoyer, L., Gool, L. V., & Tombari, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72664-4_3 · Cited by 25
Wu, J., Li, X., Ding, H., Li, X., Cheng, G., Tong, Y., & Loy, C. C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02005 · Cited by 21
3SHNet: Boosting image–sentence retrieval via visual semantic–spatial self-highlighting
Ge, X., Xu, S., Chen, F., Wang, J., Wang, G., An, S., & Jose, J. M. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103716 · Cited by 22
“Although our approach is relatively lower than ALIGN (Jia et al., 2021), which has stronger computation facilities and a larger number of image-text pairs at larger batch sizes, we propose a scheme that can apply our proposed method on large-scale datasets, which we will explore further in future work.”
InfluentialUniChest: Conquer-and-Divide Pre-Training for Multi-Source Chest X-Ray Classification
Dai, T., Zhang, R., Hong, F., Yao, J., Zhang, Y., & Wang, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3381123 · Cited by 31
“Recently, Vision-Language Pre-training (VLP) models have achieved impressive success in natural domain [30], [31], [32], which then drives many extensions in the medical area and improves the ability of machine-aided medical applications.”
BackgroundOpen-Vocabulary 3D Semantic Segmentation with Foundation Models
Li, J., Shi, S., & Schiele, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02011 · Cited by 34
“Vision-language models like CLIP [46] and ALIGN [20] align image and text features, excelling in zero-shot tasks due to extensive paired training data.”
BackgroundFusDreamer: Label-Efficient Remote Sensing World Model for Multimodal Data Classification
Wang, J., Song, W., Chen, H., Ren, J., & Zhao, H. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3554862 · Cited by 22
“…encoding image and text features and then using a dot product or multilayer perception to capture their interactions and obtain high-level representations of the modalities, such as a contrastive language–image pre-training (CLIP) [41] and a large-scale image and noisy-text embedding (ALIGN) [42].”
Mind the modality gap: Towards a remote sensing vision-language model via cross-modal alignment
Zavras, A., Michail, D., Demir, B., & Papoutsis, I. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.06.019 · Cited by 30
Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching
Biten, A. F., Mafla, A., Gómez, L., & Karatzas, D. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00254 · Cited by 21
Revisiting a kNN-Based Image Classification System with High-Capacity Storage
Nakata, K., Ng, Y., Miyashita, D., Maki, A., Lin, Y.-C., & Deguchi, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_26 · Cited by 24
Mañas, O., Rodríguez, P., Ahmadi, S., Nematzadeh, A., Goyal, Y., & Agrawal, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.185 · Cited by 45
Réby, K., Guilhelm, A., & Luca, L. D. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00184 · Cited by 24
ProbVLM: Probabilistic Adapter for Frozen Vison-Language Models
Upadhyay, U., Karthik, S., Mancini, M., & Akata, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00182 · Cited by 23
LoGoPrompt: Synthetic Text Images Can Be Good Visual Prompts for Vision-Language Models
Cheng, S., & Yang, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00274 · Cited by 30
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H. S., Zhang, X.-P. S., & Tang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01282 · Cited by 28
Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning
Kang, W., Mun, J., Lee, S., & Roh, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00275 · Cited by 24
DeTAL: Open-Vocabulary Temporal Action Localization With Decoupled Networks
Li, Z., Zhong, Y., Song, R., Li, T., Ma, L., & Zhang, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3395778 · Cited by 21
Self-citationDenseCLIP: Extract Free Dense Labels from CLIPPreprint
Zhou, C., Loy, C. C., & Dai, B. (2021) · arXiv (Cornell University) · Cited by 21
Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation
Li, X.-Y., Li, Y.-K., Du, Z.-K., Li, F.-L., Lu, K., & Li, J. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.02205 · Cited by 33
Taming Self-Training for Open-Vocabulary Object Detection
Zhao, S., Schulter, S., Zhao, L., Zhang, Z.-X., Kumar, B. G. V., Suh, Y., Chandraker, M. K., & Metaxas, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01322 · Cited by 22
Emergent Open-Vocabulary Semantic Segmentation from Off-the-Shelf Vision-Language Models
Luo, J., Khandelwal, S., Sigal, L., & Li, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00386 · Cited by 21
“Architecturally, the straightforward approach to training such methods involves aligning visual and textual latent representations via a simple dot product [24, 49].”
MethodAbdulgalil, H. D., & Basir, O. A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100159 · Cited by 20
The World of an Octopus: How Reporting Bias Influences a Language Model’s Perception of Color
Paik, C., Aroca-Ouellette, S., Roncone, A., & Kann, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.63 · Cited by 28
Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval
Liao, L., Yang, M., & Zhang, B. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3203247 · Cited by 21
Understanding image-text relations and news values for multimodal news analysis
Cheema, G. S., Hakimov, S., Müller-Budack, E., Otto, C., Bateman, J. A., & Ewerth, R. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1125533 · Cited by 32
X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation
van Sonsbeek, T., & Worring, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34048-2_36 · Cited by 24
FashionSAP: Symbols and Attributes Prompt for Fine-Grained Fashion Vision-Language Pre-Training
Han, Y., Zhang, L., Chen, Q., Chen, Z., Li, Z., Yang, J., & Cao, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01443 · Cited by 22
Self-citationMixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering
Jiang, J., & Zheng, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02318 · Cited by 24
SViTT: Temporal Learning of Sparse Video-Text Transformers
Li, Y., Min, K., Tripathi, S., & Vasconcelos, N. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01814 · Cited by 20
Affection: Learning Affective Explanations for Real-World Visual Data
Achlioptas, P., Ovsjanikov, M., Guibas, L. J., & Tulyakov, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00642 · Cited by 22
Actor-agnostic Multi-label Action Recognition with Multi-modal Query
Mondal, A., Nag, S., Prada, J. M., Zhu, X., & Dutta, A. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00086 · Cited by 22
Equivariant Similarity for Vision-Language Foundation Models
Wang, T., Lin, K., Li, L., Lin, C.-C., Yang, Z., Zhang, H., Liu, Z., & Wang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01102 · Cited by 32
Density-invariant Features for Distant Point Cloud Registration
Liu, Q., Zhu, H., Zhou, Y., Li, H., Chang, S., & Guo, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01670 · Cited by 20
Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation
Xu, W., Xu, R., Wang, C., Xu, S., Guo, L., Zhang, M., & Zhang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28456 · Cited by 20
CLIM: Contrastive Language-Image Mosaic for Region Representation
Wu, S., Zhang, W., Xu, L., Jin, S., Liu, W., & Loy, C. C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28428 · Cited by 24
Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation
Hu, X., Zhang, C., Zhang, Y. H., Hai, B., Yu, K., & He, Z. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00550 · Cited by 28
“VLMs are pre-trained on massive amounts of pairs of texts and images available on the Internet, allowing them to gain a deeper understanding of the connection between language and vi-* sion and learn more comprehensive visual representations [16,31].”
BackgroundMethodInfluentialLP-OVOD: Open-Vocabulary Object Detection by Linear Probing
Pham, C., Vu, T., & Nguyen, K. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00083 · Cited by 20
“To address this challenge, a pretrained visual-text embedding model, such as CLIP [28] or ALIGN [15], is provided as a joint text-image embedding where base and novel classes co-exist.”
MethodOpen-Vocabulary Animal Keypoint Detection with Semantic-Feature Matching
Zhang, H., Xu, L., Lai, S., Shao, W., Zheng, N., Luo, P., Qiao, Y., & Zhang, K. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02126-3 · Cited by 26
“The advent of vision-language models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) has underscored their potential in tasks that require simultaneous processing of visual and text data, ideal for open-world learning scenarios.”
Rashid, M. B., Rahaman, M. S., & Rivas, P. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030074 · Cited by 23
“Jia et al. [78] introduced Large-Scale Image and Noisy-Text (ALIGN), where they used EfficientNet for image coding, a variation in CNN architecture.”
MethodPartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation
Umam, A., Yang, C.-K., Chen, M.-H., Chuang, J., & Lin, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00333 · Cited by 29
“Based on learning granularity, vision-language models (VLMs) can be grouped into three categories, including the image-level [15, 37], pixel-level [21, 27, 53], and object-level [22, 25, 46] categories.”
BackgroundSelf-prompting semantic segmentation of bridge point cloud data using a large computer vision model
Cui, N., Chen, H., Guo, X., Zeng, Y., Hua, Z., Xiong, G., Yue, R., & Liu, J. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105729 · Cited by 20
Architectural Synergies in Bi-Modal and Bi-Contrastive Learning
Gu, Y., Liu, B., Zhang, T., Sha, X., & Chen, S. (2024) · IEEE Access · doi:10.1109/access.2024.3457586 · Cited by 19
Bianchi, L., Carrara, F., Messina, N., Gennaro, C., & Falchi, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02125 · Cited by 20
“However, recent advancements in large-scale image-text encoder models — such as CLIP [29] and ALIGN [11], trained on millions of image-text pairs — enabled a strong semantic interaction between vision and language.”
BackgroundSHiNe: Semantic Hierarchy Nexus for Open-Vocabulary Object Detection
Liu, M., Hayes, T. L., Ricci, E., Csurka, G., & Volpi, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01574 · Cited by 23
“Prompt engineering [17] has been extensively studied as a technique to enhance VLMs [22, 42, 66].”
BackgroundMethodTextual Query-Driven Mask Transformer for Domain Generalized Segmentation
Pak, B., Woo, B., Kim, S., Kim, D., & Kim, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_3 · Cited by 24
Vision-Language Action Knowledge Learning for Semantic-Aware Action Quality Assessment
Xu, H., Ke, X., Li, Y., Xu, R., Wu, H., Lin, X., & Guo, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_24 · Cited by 20
Adversarial Prompt Tuning for Vision-Language Models
Zhang, J., Ma, X., Wang, X., Qiu, L., Wang, J., Jiang, Y., & Sang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72995-9_4 · Cited by 27
DIAL: Dense Image-Text ALignment for Weakly Supervised Semantic Segmentation
Jang, S., Yun, J., Kwon, J., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72890-7_15 · Cited by 19
Ge, J., Zhang, X., Zheng, Y., Guo, K., & Liang, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.05.002 · Cited by 22
“CoCa (Yu et al., 2022) used the ALIGN dataset as well as a private dataset called JFT-3B, containing 3 billion image-text pairs, to train a ViT-g / 14 visual encoder and a language model.”
InfluentialFoundation models and Transformers for anomaly detection: A survey
Ammar, M. B., Mendoza, A., Belkhir, N., Antoine Manzanera, B., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103517 · Cited by 25
Liu, H., Liu, S., An, R., Zhuo, Y., & Tao, J. (2023) · Mathematics · doi:10.3390/math11102279 · Cited by 20
Unsupervised Video Anomaly Detection Based on Similarity with Predefined Text Descriptions
Kim, J. H., Yoon, S., Choi, T., & Sull, S. (2023) · Sensors · doi:10.3390/s23146256 · Cited by 21
mCLIP: Multilingual CLIP via Cross-lingual Transfer
Chen, G., Hou, L., Chen, Y., Dai, W., Shang, L., Jiang, X., Liu, Q., Pan, J., & Wang, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.728 · Cited by 23
Self-citationFew-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment
Wang, R., Zheng, H. R., Duan, X., Liu, J., Lu, Y., Wang, T., Xu, S., & Zhang, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02245 · Cited by 21
Foundation Model Drives Weakly Incremental Learning for Semantic Segmentation
Yu, C., Zhou, Q., Li, J., Yuan, J., Wang, Z., & Wang, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02268 · Cited by 22
Multi-Granularity Matching Transformer for Text-Based Person Search
Bao, L., Wei, L., Zhou, W., Liu, L., Xie, L., Li, H., & Tian, Q. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321504 · Cited by 19
Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
Wu, C.-E. R., Tian, Y., Yu, H., Wang, H., Morgado, P., Hu, Y. H., & Yang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01420 · Cited by 24
Open-Vocabulary Object Detection With an Open Corpus
Wang, J., Zhang, H., Hong, H., Jin, X., He, Y., Xue, H., & Zhao, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00622 · Cited by 21
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
