The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 901–1,000 of 1,011
Bridging the Cross-Modality Semantic Gap in Visual Question Answering
Wang, B., Ma, Y., Li, X., Gao, J., Hu, Y., & Yin, B. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3370925 · Cited by 13
“Contrastive learning [40], [41], [42], [43] aims to enhance the quality of learned representations by maximizing the mutual information between input samples and their corresponding similar samples.”
Bridging the Semantic Latent Space between Brain and Machine: Similarity Is All You Need
Chen, J., Qi, Y., Wang, Y., & Pan, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29009 · Cited by 14
“Recently, large-scale image-text pre-training, represented by CLIP (Radford et al. 2021) and ALIGN (Jia et al. 2021), is encouraging, which provides a new light on the zero-shot transfer.”
BackgroundCombinatorial discovery of antibacterials via a feature-fusion based machine learning workflow
Wang, C., Wu, Y., Xue, Y., Zou, L., Huang, Y., Zhang, P., & Ji, J. (2024) · Chemical Science · doi:10.1039/d3sc06441g · Cited by 12
PGVT: Pose-Guided Video Transformer for Fine-Grained Action Recognition
Zhang, H., Leong, M. C., Li, L., & Lin, W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00651 · Cited by 13
“Recent works [28,39,48, 71, 83, 92, 102, 110] have utilized large-scale multi-modal datasets, such as image / video-text pairs, to train models, resulting in robust visual representations.”
BackgroundImage Labels Are All You Need for Coarse Seagrass Segmentation
Raine, S., Marchant, R., Kusý, B., Maire, F., & Fischer, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00584 · Cited by 13
Foundational Models for Pathology and Endoscopy Images: Application for Gastric Inflammation
Kerdegari, H., Higgins, K., Veselkov, D. A., Laponogov, I., Poļaka, I., Coimbra, M. T., Pescino, J. A., Leja, M., Dinis-Ribeiro, M. J., Fleitas, T., et al. (2024) · Diagnostics · doi:10.3390/diagnostics14171912 · Cited by 14
Zhang, Y., Luo, H., & Lei, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01241 · Cited by 12
“With the advent of large image-language models [15, 20–22, 32, 33], researchers [19, 26, 45] begin to develop novel implicit language-free training methods.”
BackgroundAny-Shift Prompting for Generalization Over Distributions
Xiao, Z., Shen, J., Derakhshani, M. M., Liao, S., & Snoek, C. G. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01314 · Cited by 20
“Image-language foundation models such as CLIP [52] and ALIGN [25] achieve significant advances in various downstream tasks.”
Learning to Adapt SAM for Segmenting Cross-Domain Point Clouds
Peng, X., Chen, R., Qiao, F., Kong, L., Liu, Y., Sun, Y., Wang, T., Zhu, X., & Ma, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72775-7_4 · Cited by 27
“The rise of foundation models [12, 21, 39] has garnered significant attention which are trained on extensive datasets, consequently demonstrating exceptional performance.”
BackgroundLanguage-Assisted Skeleton Action Understanding for Skeleton-Based Temporal Action Segmentation
Ji, H., Chen, B., Xu, X., Ren, W., Wang, Z., & Liu, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72949-2_23 · Cited by 13
Improving Zero-Shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
Mistretta, M., Baldrati, A., Bertini, M., & Bagdanov, A. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72907-2_27 · Cited by 16
Evolving Interpretable Visual Classifiers with Large Language Models
Chiquier, M., Mall, U., & Vondrick, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73039-9_11 · Cited by 24
Text-guided Image Restoration and Semantic Enhancement for Text-to-Image Person Retrieval
Liu, D., Li, H., Zhao, Z., & Dong, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107028 · Cited by 13
SenCLIP: Enhancing Zero-Shot Land-Use Mapping for Sentinel-2 with Ground-Level Prompting
Jain, P., Ienco, D., Interdonato, R., Berchoux, T., & Marcos, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00552 · Cited by 17
“Prominent examples of VLMs include CLIP [28], ALIGN [12], and BLIP [16, 17], which aim to map both textual descriptions and visual representations into the same latent space, enabling direct semantic comparisons between the two modalities.”
Toward a Holistic Evaluation of Robustness in CLIP Models
Tu, W., Deng, W., & Gedeon, T. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3580234 · Cited by 18
“Contemporary research has delved into multiple facets of CLIP models.”
PreDet: Large-scale weakly supervised pre-training for detection
Ramanathan, V., Wang, R., & Mahajan, D. K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00286 · Cited by 12
NAPReg: Nouns As Proxies Regularization for Semantically Aware Cross-Modal Embeddings
Jawade, B., Mohan, D. D., Ali, N. M., Setlur, S., & Govindaraju, V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00119 · Cited by 13
Ma, Z., Yu, Z., Li, J., & Li, G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26569 · Cited by 12
Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-Commerce
Jin, Y., Li, Y., Yuan, Z., & Mu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01064 · Cited by 12
Meta-Personalizing Vision-Language Models to Find Named Instances in Video
Yeh, C.-H., Russell, B. C., Šivic, J., Heilbron, F. C., & Jenni, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01833 · Cited by 12
Text Augmented Spatial Aware Zero-shot Referring Image Segmentation
Suo, Y., Zhu, L., & Yang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.73 · Cited by 22
Self-citationCiT: Curation in Training for Effective Vision-Language Data
Hu, X., Xie, S., Huang, P.-Y., Yu, L., Howes, R., Ghosh, G., Zettlemoyer, L., & Feichtenhofer, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01393 · Cited by 26
Multi-modal recursive prompt learning with mixup embedding for generalization recognition
Jia, Y., Ye, X., Liu, Y., & Guo, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111726 · Cited by 12
Homogeneous vector bundles and G-equivariant convolutional neural networks
Aronsson, J. (2022) · Sampling Theory Signal Processing and Data Analysis · doi:10.1007/s43670-022-00029-3 · Cited by 12
Modeling Text-visual Mutual Dependency for Multi-modal Dialog GenerationPreprint
Wang, S., Meng, Y., Sun, X., Wu, F. Q., Ouyang, R., Yan, R., Zhang, T., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14445 · Cited by 12
MURAL: Multimodal, Multitask Retrieval Across LanguagesPreprint
Jain, A., Guo, M., Srinivasan, K., Chen, T., Kudugunta, S., Jia, C., Yang, Y., & Baldridge, J. (2021) · arXiv.org · doi:10.48550/arxiv.2109.05125 · Cited by 65
Self-citationVideo Foundation Models for Animal Behavior AnalysisPreprint
Sun, J. J., Zhou, H., Zhao, L., Yuan, L., Seybold, B., Hendon, D., Schroff, F., Ross, D. A., Adam, H., Hu, B., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.30.605655 · Cited by 16
“These models employ transformers specifically trained to align web-scale image-text paired data through contrastive learning [4, 29], predicting the next token in the language modality [30] or in the interleaved image-text sequence [31].”
BackgroundMethodBulat, A., & Tzimiropoulos, G. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01904-9 · Cited by 12
“…large scale V&L pre-training with contrastive learning has been used to train foundation models resulting in robust representations, transferable to new tasks both under few-shot and zero-shot settings (Radford et al., 2021; Jia et al., 2021; Li et al., 2021; Yao et al., 2021; Yu et al., 2022).”
InfluentialExploiting multi-scale contextual prompt learning for zero-shot semantic segmentation
Wang, Y., & Tian, Y. (2023) · Displays · doi:10.1016/j.displa.2023.102616 · Cited by 11
A Retrospect to Multi-prompt Learning across Vision and Language
Chen, Z., Huang, X., Guan, Q., Lin, L., & Luo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02028 · Cited by 11
DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability
Huang, R., Han, J., Lu, G., Liang, X., Zeng, Y., Zhang, W., & Xu, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01440 · Cited by 15
Learning to Learn Better Visual Prompts
Wang, F., Huang, W., Yang, S., Fan, Q., & Lan, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28343 · Cited by 12
“…fusion encoder (single-stream (Su et al. 2019; Li et al. 2020, 2019) and dual-stream (Li et al. 2021; Lu et al. 2019; Tan, Bansal, and Assoc Com-putat 2019)), dual encoder (Lee et al. 2018; Jia et al. 2021; Radford et al. 2021) and the combination of both (Bao et al. 2022; Singh et al. 2022).”
BackgroundFSODv2: A Deep Calibrated Few-Shot Object Detection Network
Fan, Q., Zhuo, W., Tang, C., & Tai, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02049-z · Cited by 11
Vision-language pre-training via modal interaction
Cheng, H., Ye, H., Zhou, X., Liu, X., Chen, F., & Wang, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110809 · Cited by 12
Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data
Wang, A., Dai, S., Wu, H., & Iwahori, Y. (2024) · Remote Sensing · doi:10.3390/rs16163082 · Cited by 11
“Despite the significant advances in classification performance achieved by large-scale image-text pre-training techniques, some inherent limitations remain [41].”
Generalization Boosted Adapter for Open-Vocabulary Segmentation
Xu, W., Wang, C., Feng, X., Xu, R., Huang, L., Zhang, Z., Guo, L., & Xu, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3454227 · Cited by 13
“Recently, vision-language models (VLMs) [11], [12] have gained significant attention due to their remarkable open-vocabulary object recognition capability.”
Low-Resource Vision Challenges for Foundation Models
Zhang, Y., Doughty, H., & Snoek, C. G. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02073 · Cited by 16
“An intuitive way to handle low-resource vision is leveraging the strong image representations from foundation models, which have progressed at a tremendous pace in recent years [25, 37, 42, 47, 55].”
BackgroundSynthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
Peng, W., Xie, S., You, Z., Lan, S., & Wu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01261 · Cited by 16
“Models such as CLIP [23], ALIGN [13], CyCLIP [10] CoCa [38], Om-niVL [31] and Open-VCLIP [33, 34] have demonstrated impressive performance across a wide range of downstream tasks.”
BackgroundMethodPeVL: Pose-Enhanced Vision-Language Model for Fine-Grained Human Action Recognition
Zhang, H., Leong, M. C., Li, L., & Lin, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01784 · Cited by 11
“The details of the novel designs for each block are described in the technical part of the main paper.”
Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning
Zhang, W., Wan, C., Liu, T., Tian, X., Shen, X., & Ye, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01751 · Cited by 13
“Recent years have witnessed remarkable achievements in contrastive language-image pre-training models [9, 25, 36, 60, 87, 88], with CLIP [60] emerging as the front-runner.”
Toubal, I. E., Avinash, A., Alldrin, N., Dlabal, J., Zhou, W., Luo, E., Stretcu, O., Xiong, H., Lu, C.-T., Zhou, H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01662 · Cited by 15
“Specifically, we first extract image features from a foundation vision model (CLIP or ALIGN) [23, 24].”
BackgroundMethodSelf-citationSegIC: Unleashing the Emergent Correspondence for In-Context Segmentation
Meng, L., Lan, S., Li, H., Álvarez, J. M., Wu, Z., & Jiang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72920-1_12 · Cited by 15
Multimodal Fake News Detection Based on Contrastive Learning and Similarity Fusion
Li, Y., Jia, K., & Wang, Q. (2024) · IEEE Access · doi:10.1109/access.2024.3481311 · Cited by 14
Dataset Enhancement with Instance-Level Augmentations
Kupyn, O., & Rupprecht, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73337-6_22 · Cited by 17
Cascade Prompt Learning for Vision-Language Model Adaptation
Wu, G., Zhang, X., Li, Z., Chen, Z., Liang, J., Yang, J. J., & Li, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_18 · Cited by 16
“Foundational Vision-Language Models (VLMs) like CLIP [43] and ALIGN [19] have demonstrated significant advancements in recent years across a diverse spectrum of tasks [6,10,12,38,57,60,63].”
BackgroundSelf-citationRobust Calibration of Large Vision-Language Adapters
Murugesan, B., Silva-Rodríguez, J., Ayed, I. B., & Dolz, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72691-0_9 · Cited by 14
Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning
Singha, M., Jha, A., Gupta, D., Singla, P., & Banerjee, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72691-0_1 · Cited by 11
EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrieval
Hummel, T., Karthik, S., Georgescu, M.-I., & Akata, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72913-3_1 · Cited by 18
Multimodal large model pretraining, adaptation and efficiency optimization
Ji, L., Xiao, S., Feng, J., Gao, W., & Zhang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129138 · Cited by 14
MKFTracker: An RGBT tracker via multimodal knowledge embedding and feature interaction
Li, F., Xia, W., Zhou, D., & Cao, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112860 · Cited by 11
Learning with Enriched Inductive Biases for Vision-Language Models
Yang, L., Zhang, R., Chen, Q., & Xie, X. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02354-1 · Cited by 14
“Recently, Vision-Language Models (VLMs) (Radford et al., 2021; Jia et al., 2021; Yao et al., 2021; Yuan et al., 2021; Zhai et al., 2022; Huang et al., 2023; Wang et al., 2021; Alayrac et al., 2022) have emerged to reduce the need of manually collecting such high-quality annotations.”
InfluentialEff-SAM: SAM-Based Efficient Method for Brain Tumor Segmentation in Multimodal 3D MRI Scans
Ahmad, N., & Chen, Y.-T. (2025) · IEEE Access · doi:10.1109/access.2025.3571464 · Cited by 14
Self-citationTransformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 116
Trustworthy Visual-Textual Retrieval
Qin, Y., Huang, L., Peng, D., Jiang, B., Zhou, J. T., Peng, X., & Hu, P. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3587575 · Cited by 12
DMFN: A disentangled multi-level fusion network for review helpfulness prediction
Ren, G., Diao, L., & Kim, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120344 · Cited by 12
Audio–text retrieval based on contrastive learning and collaborative attention mechanism
Hu, T., Xiang, X., Qin, J., & Tan, Y. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01144-4 · Cited by 11
Learning to Name Classes for Vision and Language Models
Parisot, S., Yang, Y., & McDonagh, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02248 · Cited by 12
Self-citationMuthusamy, V., Rizk, Y., Kate, K., Venkateswaran, P., Isahagian, V., Gulati, A., & Dube, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.461 · Cited by 17
Agile Modeling: From Concept to Classifier in Minutes
Stretcu, O., Vendrow, E. B., Hata, K., Viswanathan, K., Ferrari, V., Tavakkol, S., Zhou, W., Avinash, A., Luo, E., Alldrin, N., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02040 · Cited by 13
OmniLabel: A Challenging Benchmark for Language-Based Object Detection
Schulter, S., G, V. K. B., Suh, Y., Dafnis, K. M., Zhang, Z., Zhao, S., & Metaxas, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01098 · Cited by 13
Rectify representation bias in vision-language models for long-tailed recognition
Li, B., Yao, Y., Tan, J., Gong, R., Lu, J., & Luo, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106134 · Cited by 13
Mutual-Modality Adversarial Attack with Semantic Perturbation
Ye, J., Yu, R., Liu, S., & Wang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28488 · Cited by 11
Mix-tower: Light visual question answering framework based on exclusive self-attention mechanism
Chen, D., Chen, J., Yang, L., & Shang, F. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127686 · Cited by 11
Prompt-Guided Semantic-Aware Distillation for Weakly Supervised Incremental Semantic Segmentation
Hao, X., Jiang, X., Ni, W., Tan, W., & Yan, B. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3412996 · Cited by 13
“Among the successes of large pre-trained vision-language models [69], [70], CLIP [69] has been pre-trained on large-scale image-text pairs from the web in a contrastive manner, showing promising generality.”
An Analysis of Semantically-Aligned Speech-Text Embeddings
Huzaifah, M., & Kukanov, I. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023147 · Cited by 11
Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles
Ye, S., Xie, Y., Chen, D., Yi‐chong, X., Yuan, L., Zhu, C., & Liao, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00259 · Cited by 16
CLIPScore: A Reference-free Evaluation Metric for Image CaptioningPreprint
Hessel, J., Holtzman, A., Forbes, M., Bras, R. J. L., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08718 · Cited by 1,051
A Simple Long-Tailed Recognition Baseline via Vision-Language ModelPreprint
Ma, T., Geng, S., Wang, M., Shao, J., Lu, J., Li, H., Gao, P., & Qiao, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14745 · Cited by 57
Preface: Advancing deep learning for remote sensing time series data analysis
Zhang, H. K., Camps‐Valls, G., Liang, S., Tuia, D., Pelletier, C., & Zhu, Z. (2025) · Remote Sensing of Environment · doi:10.1016/j.rse.2025.114711 · Cited by 13
Text-Guided Face Recognition using Multi-Granularity Cross-Modal Contrastive Learning
Hasan, M. M., Sami, S. M., & Nasrabadi, N. M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00568 · Cited by 10
“[21] designed a method called noisy student training, which utilized noisy text data to improve image-text representation learning by aligning image-text pairs using a contrastive loss.”
BackgroundMethodInfluentialZhang, K., Wu, F. H., Zhang, G., Liu, J., & Li, M. (2024) · Displays · doi:10.1016/j.displa.2024.102710 · Cited by 12
Liu, Q., & Wang, X. (2024) · Remote Sensing · doi:10.3390/rs16132289 · Cited by 11
TagCLIP: Improving Discrimination Ability of Zero-Shot Semantic Segmentation
Li, J., Chen, P., Qian, S., Liu, S., & Jia, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3454647 · Cited by 21
DFCNet +: Cross-modal dynamic feature contrast net for continuous sign language recognition
Feng, Y., Chen, N., Wu, Y., Jiang, C., Liu, S., & Chen, S. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105260 · Cited by 9
Noise-Robust Vision-Language Pre-Training With Positive-Negative Learning
Huang, Z., Yang, M., Xiao, X., Hu, P., & Peng, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3462996 · Cited by 15
MoDE: CLIP Data Experts via Clustering
Ma, J., Huang, P.-Y., Xie, S., Li, S.-W. D., Zettlemoyer, L., Chang, S., Yih, W.-T., & Xu, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02489 · Cited by 15
“Scaling up [17, 34] existing approaches and improving the effectiveness is critical.”
BackgroundHyperbolic Learning with Synthetic Captions for Open-World Detection
Kong, F., Chen, Y., Cai, J., & Modolo, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01586 · Cited by 11
“VLMs have demonstrated remarkable abilities in tasks like image classification and image-text retrieval [7, 19, 41, 52, 55, 61].”
BackgroundSelf-citationGao, T., Ao, W., Wang, X.-A., Zhao, Y., Ma, P., Xie, M., Fu, H., Ren, J., & Gao, Z. Y. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00283 · Cited by 10
“Recently, the surge of vision-language models such as CLIP [15] and ALIGN [16] has greatly boosted zero-shot learning.”
BackgroundLearning Representations of Satellite Images From Metadata Supervision
Bourcier, J., Dashyan, G., Alahari, K., & Chanussot, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73383-3_4 · Cited by 13
SwiftBrush V2: Make Your One-Step Diffusion Model Better Than Its Teacher
Dao, T., Nguyen, T. H., Le, T. C., Vu, D. T. S., Nguyen, K., Pham, C., & Tran, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73007-8_11 · Cited by 10
Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classification
Gao, Y., Gu, D., Zhou, M., & Metaxas, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72117-5_5 · Cited by 10
Deep Residual Coupled Prompt Learning for Zero-Shot Sketch-Based Image Retrieval
Zhuo, G., Shu, Z., & Yu, Z. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3481898 · Cited by 10
CLAP: Isolating Content from Style Through Contrastive Learning with Augmented Prompts
Cai, Y., Liu, Y., Zhang, Z., & Shi, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72664-4_8 · Cited by 12
“Vision-language models, exemplified by CLIP [36], have garnered substantial attention due to their exceptional generalization capabilities, achieved through the learned features, obtained by utilizing a cross-modal contrastive loss [20,25,36].”
Yu, Y.-C., Huang, C.-P., Chen, J. C., Chang, K.-P., Lai, Y.-H., Yang, F.-E., & Wang, Y.-C. F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73347-5_13 · Cited by 12
Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
Xu, Y., Zhang, M., Yang, X., & Xu, C. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3485518 · Cited by 17
LAPT: Label-Driven Automated Prompt Tuning for OOD Detection with Vision-Language Models
Zhang, Y., Zhu, W., He, C., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73220-1_16 · Cited by 16
Cheng, H., Xiao, E., Gu, J., Yang, L., Duan, J., Zhang, J., Cao, J., Xu, K., & Xu, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73202-7_11 · Cited by 25
Zhang, H., & Zhang, R. (2024) · Journal of Infrastructure Intelligence and Resilience · doi:10.1016/j.iintel.2024.100137 · Cited by 12
DST-Det: Open-Vocabulary Object Detection via Dynamic Self-Training
Xu, S., Li, X., Wu, S., Zhang, W., Tong, Y., & Loy, C. C. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3520734 · Cited by 26
“, 2023b) has drawn increasing attention due to the emergence of vision-language models (Radford et al., 2021; Jia et al., 2021).”
InfluentialChen, Y., He, T. C., Fu, J., Wang, L., Guo, J., Hu, T., & Cheng, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521718 · Cited by 14
“Inspired by their success, foundational vision-language models such as CLIP [9] and ALIGN [26] have gained widespread adoption in the domain of RGB-based action recognition [1], [10], [11], [27], [28], [29], [30], [31], [32].”
Self-citationTowards a Multimodal Framework for Remote Sensing Image Change Retrieval and Captioning
Ferrod, R., Di, L., & Ienco, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78980-9_15 · Cited by 10
GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
Jeon, B., Wu, M., Cao, S., Kim, S., Park, S., Aggarwal, N., Unger, C., Arfeen, D., Liao, P., Miao, X., et al. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707220 · Cited by 23
“Table 1 presents the search times by the three optimizers (GraphPipe, PipeDream, and Piper) for the three models (Multi-Modal Transformer, DLRM, and CANDLE-Uno).”
InfluentialZhang, W., Zhang, Y., Lin, J., Huang, B., Zhang, J., & Yu, W. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111547 · Cited by 13
Ambiguity-Aware and High-order Relation learning for multi-grained image–text matching
Chen, J., Gao, Y., Ge, M., & Li, M. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113355 · Cited by 10
“Previous image-text matching tasks have predominantly employed self-supervised contrastive learning frameworks [16, 17, 23, 32, 61, 62], utilizing objective functions such as InfoNCE loss [63] and Hard Negative Triplet loss [14].”
Active Learning for Vision-Language Models
Safaei, B., & Patel, V. M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00480 · Cited by 11
“However, recent advancements in VLP, exemplified by models like CLIP [38] and ALIGN [19] models, have achieved impressive outcomes.”
DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models
Ali, E., Silva, S., & Khan, M. H. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00593 · Cited by 13
“Large-scale Vision-Language (VL) Models: Among other vision-language models [6, 8, 16, 20, 38, 41, 56], CLIP [38] stands out as a pioneering example, aligning visual and textual features via a contrastive objective on a vast web-crawled collection of image-text pairs.”
Language–Image Consistency Augmentation and Distillation Network for visual grounding
Ke, X., Xu, P., & Guo, W. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111663 · Cited by 10
Wang, G., Xie, J., Zhang, T., Sun, Y., Chen, H., Zhuang, Y., & Li, J. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3564332 · Cited by 15
Bimodal speech emotion recognition via contrastive self-alignment learning
Wang, C., Wen, G., Yang, P., & Liu, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128605 · Cited by 11
A review of few-shot image classification: Approaches, datasets and research trends
Lim, J. Y., Lim, K. M., Lee, C. P., & Tan, Y. X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130774 · Cited by 18
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
