The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 701–800 of 1,011
Shatter and Gather: Learning Referring Image Segmentation with Text Supervision
Kim, D.-W., Kim, N., Lan, C., & Kwak, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01425 · Cited by 21
DIME-FM : DIstilling Multimodal and Efficient Foundation Models
Sun, X., Zhang, P., Zhang, P., Shah, H., Saenko, K., & Xia, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01423 · Cited by 24
CoPL: Contextual Prompt Learning for Vision-Language Understanding
Goswami, K., Karanam, S., Udhayanan, P., Joseph, K. J., & Srinivasan, B. V. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29766 · Cited by 25
Domain Aligned CLIP for Few-shot Classification
Gondal, M. W., Gast, J., Ruiz, I. A., Droste, R., Macri, T., Kumar, S., & Staudigl, L. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00562 · Cited by 20
“The growing popularity of large-scale representation learning models for multi-modal data like CLIP [33], ALIGN [20] and Florence [50] has highlighted the need for efficiently adapting these models to various downstream tasks across multiple domains and applications.”
BackgroundKaul, P., Li, Z., Yang, H., Dukler, Y., Swaminathan, A. D., Taylor, C. J., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02571 · Cited by 37
“In contrast to earlier vision-language models such as CLIP [18, 38], LVLMs are generally comprised of a pretrained LLM and image encoder, aligned with a connector module of varying complexity.”
Self-citationBootstrapping Autonomous Driving Radars with Self-Supervised Learning
Hao, Y., Madani, S., Guan, J., Alloulah, M., Gupta, S., & Hassanieh, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01422 · Cited by 24
“…whereby paired positive views from other modalities replace or complement augmentation in vision SSL. Examples include vision and sound [2, 5– 9, 44, 53], vision and text [36, 59], different formats of medical imaging [69], vision and point clouds [1, 33, 73], and vision and radar [3, 4, 35, 54].”
BackgroundPracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization
Chen, Z., Wang, W., Zhao, Z., Su, F., Men, A., & Meng, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02218 · Cited by 22
“Recently, Vision-Language Models (VLMs) have shown powerful zero-shot transfer ability [22, 28, 38] on various downstream tasks.”
BackgroundA Simple Recipe for Contrastively Pre-Training Video-First Encoders Beyond 16 Frames
Papalampidi, P., Koppula, S., Pathak, S., Chiu, J., Heyward, J., Pătrăucean, V., Shen, J., Miech, A., Zisserman, A., & Nematzdeh, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01364 · Cited by 29
“Following common practice [1, 30], we use a dual vision– language architecture with a Noise Contrastive Estimation (NCE) loss [17, 45, 65] to pre-train our vision encoder, similar to CLIP [50], ALIGN [26] and VideoCLIP [66].”
MethodInsect-Foundation: A Foundation Model and Large-Scale 1M Dataset for Visual Insect Understanding
Nguyen, H.-Q., Truong, T.-D., Nguyen, X.-B., Dowling, A. P. G., Li, X., & Luu, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02072 · Cited by 28
“Common approaches learned alignment be-tween vision and language, for example, CLIP [36], ALIGN [18], CoCa [56], to model visual concepts and data distributions.”
BackgroundMethodInfluentialProFD: Prompt-Guided Feature Disentangling for Occluded Person Re-Identification
Cui, C., Huang, S., Song, W., Ding, P., Zhang, M., & Wang, D. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680958 · Cited by 19
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Shvetsova, N., Kukleva, A., Hong, X., Rupprecht, C., Schiele, B., & Kuehne, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72992-8_1 · Cited by 34
VLAB: Enhancing Video Language Pretraining by Feature Adapting and Blending
He, X., Chen, S., Ma, F., Huang, Z., Jin, X., Liu, Z., Fu, D., Yang, Y., Liu, J., & Feng, J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521729 · Cited by 30
Self-citationCLIPVQA: Video Quality Assessment via CLIP
Xing, F., Li, M., Wang, Y., Zhu, G., & Cao, X. (2024) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2024.3511927 · Cited by 23
“Recent studies [25], [26] have shown the viability of the large-scale contrastive language-image pre-training (i.e., CLIP) method on computer vision applications.”
DPStyler: Dynamic PromptStyler for Source-Free Domain Generalization
Tang, Y., Wan, Y., Qi, L., & Geng, X. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521671 · Cited by 21
Wang, J., Lu, G., Xu, H., Li, Z., Xu, C., & Fu, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01044 · Cited by 21
Self-citationBRAVE: Broadening the Visual Encoding of Vision-Language Models
Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Zamir, A., & Tombari, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_7 · Cited by 61
Zero-shot and few-shot multimodal plastic waste classification with vision-language models
Ranjbar, I., Ventikos, Y., & Arashpour, M. (2025) · Waste Management · doi:10.1016/j.wasman.2025.114815 · Cited by 24
Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Li, S., Du, Y., Torralba, A. B., Šivic, J., & Russell, B. C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00186 · Cited by 18
Embedding Arithmetic of Multimodal Queries for Image Retrieval
Couairon, G., Douze, M., Cord, M., & Schwenk, H. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00542 · Cited by 22
ArCo: Attention-reinforced transformer with contrastive learning for image captioning
Wang, Z., Shi, S., Zhai, Z., Wu, Y., & Yang, R. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104570 · Cited by 18
Mask-Free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations
Vibashan, V. S., Yu, N., Chen, X., Qin, C., Gao, M., Niebles, J. C., Patel, V. M., & Xu, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02254 · Cited by 22
Masked Images Are Counterfactual Samples for Robust Fine-Tuning
Xiao, Y., Tang, Z., Wei, P.-X., Liu, C., & Lin, L. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01944 · Cited by 29
IFSeg: Image-free Semantic Segmentation via Vision-Language Model
Yun, S., Park, S. H., Seo, P. H., & Shin, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00290 · Cited by 19
Learning Visual Representations via Language-Guided Sampling
Banani, M. E., Desai, K. A., & Johnson, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01841 · Cited by 20
Wei, Y., Cao, Y., Zhang, Z., Peng, H., Yao, Z., Xie, Z., Hu, H., & Guo, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00272 · Cited by 21
The role of artificial intelligence in crop improvement
Negus, K. L., Li, X., Welch, S. M., & Yu, J. (2024) · Advances in agronomy · doi:10.1016/bs.agron.2023.11.001 · Cited by 21
Guiding image captioning models toward more specific captions
Kornblith, S., Li, L., Wang, Z.-R., & Nguyen, T. N. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01400 · Cited by 19
Too Large; Data Reduction for Vision-Language Pre-Training
Wang, A. J., Lin, K. Q., Zhang, D. J., Lei, S. W., & Shou, M. Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00292 · Cited by 28
Jamil, A., Rehman, S. U., Mahmood, K., Villar, M. G., Prola, T. A., Díez, I. D. L. T., Samad, M. A., & Ashraf, I. (2024) · IEEE Access · doi:10.1109/access.2024.3365528 · Cited by 23
RetractedShi, Z., Landrum, E., O'Connell, A., Kian, M., Pinto-Alva, L., Shrestha, K., Zhu, X., & Matarić, M. J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31245 · Cited by 29
CoVR-2: Automatic Data Construction for Composed Video Retrieval
Ventura, L., Yang, A., Schmid, C., & Varol, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3463799 · Cited by 31
Domain Prompt Learning with Quaternion Networks
Cao, Q., Xu, Z., Chen, Y., Chao, M., & Yang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02515 · Cited by 18
“To mitigate this dependency, ALIGN [15] leverages large-scale noisy image-text pairs, achieving comparable performance through a noise-robust contrastive learning approach.”
BackgroundMethodSelf-citationAccept the Modality Gap: An Exploration in the Hyperbolic Space
Ramasinghe, S., Shevchenko, V., Avraham, G., & Ajanthan, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02574 · Cited by 22
“Most notably, CLIP [28] and ALIGN [15] train a contrastive objective which aligns text and image embeddings.”
Vision-Language Pseudo-Labels for Single-Positive Multi-Label Learning
Xing, X., Xiong, Z., Stylianou, A., et al. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00776 · Cited by 18
“Vision-Language Model (VLM) [9, 17] is a multi-modality model, using the image and its corresponding text as supervision signal to help us better understand vision-language correlation.”
MethodTF-FAS: Twofold-Element Fine-Grained Semantic Guidance for Generalizable Face Anti-spoofing
Wang, X., Zhang, K.-Y., Yao, T., Zhou, Q., Ding, S., Dai, P., & Ji, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72667-5_9 · Cited by 18
Chen, W., Liu, J., Liu, T., & Yuan, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02246-w · Cited by 24
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
Wang, W., Ren, Y., Luo, H., Li, T., Yan, C., Chen, Z., Wang, W., Li, Q., Lu, L., Zhu, X., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_27 · Cited by 64
Contrastive Attention for Automatic Chest X-ray Report GenerationPreprint
Liu, F., Yin, C., Wu, X., Ge, S., Zou, Y., Zhang, P., Zou, Y., & Sun, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06965 · Cited by 51
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationFoundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v1 · Cited by 24
MMSR: Symbolic regression is a multi-modal information fusion task
Li, Y., Liu, J., Wu, M., Yu, L., Li, W., Ning, X., Li, W., Hao, M., Deng, Y., & Wei, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102681 · Cited by 27
“Recently, models such as Contrastive Language-Image Pre-Training (CLIP) [26] and ALIGN [27] performed pre-train on noisy images and text pairs from the web using contrastive loss.”
MethodCLIP-GAN: Stacking CLIPs and GAN for Efficient and Controllable Text-to-Image Synthesis
Hou, Y., Zhang, W., Zhu, Z., & Yu, H. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535304 · Cited by 20
Liu, X., Shi, G., Wang, R., Lai, Y., Zhang, J., Han, W., Lei, M., Li, M., Zhou, X., Wu, Y., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103550 · Cited by 18
Yu, S., Dou, Z., & Wang, S. (2023) · Tsinghua Science & Technology · doi:10.26599/tst.2022.9010044 · Cited by 17
Deeply Coupled Cross-Modal Prompt Learning
Liu, X., Tang, W., Lu, J., Zhao, R., Guo, Z., & Tan, F. Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.504 · Cited by 21
DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training
Chen, Y., Qi, X., Wang, J., & Zhang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02169 · Cited by 18
Self-citationDeep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection
Yang, Y., Bao, R., Guo, W., Zhan, D., Yin, Y., & Yang, J. J. (2023) · Science China Information Sciences · doi:10.1007/s11432-021-3530-7 · Cited by 19
“For example, state-of-the-art approaches in bidirectional image-sentence retrieval [36,37] have leveraged visual-linguistic consistency to learn consistent embeddings and achieved great success on standard datasets, such as MSCOCO [38] and Flickr30K [39].”
BackgroundSelf-citationSTAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Chen, C., Zhang, B., Cao, L., Shen, J., Gunter, T., Jose, A., Toshev, A., Zheng, Y., Shlens, J., Pang, R., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.932 · Cited by 36
Self-citationVideo Action Recognition with Attentive Semantic Units
Chen, Y., Chen, D., Liu, R., Li, H., & Peng, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00933 · Cited by 20
Self-citationImproving CLIP Fine-tuning Performance
Wei, Y., Hu, H., Xie, Z., Liu, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D., & Guo, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00501 · Cited by 20
Cai, K., Ren, P., Zhu, Y., Xu, H., Liu, J., Li, C., Wang, G., & Liang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00116 · Cited by 18
Commonsense Knowledge Prompting for Few-Shot Action Recognition in Videos
Shi, Y., Wu, X., Lin, H., & Luo, J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3361157 · Cited by 20
Yang, B., Liu, F., Zou, Y., Wu, X., Wang, Y., & Clifton, D. A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3371376 · Cited by 22
Hu, D., Jiang, Z., Shi, J., Xie, F., Wu, K., Tang, K., Cao, M. W., Huai, J., & Zheng, Y. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103163 · Cited by 20
Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction
Koch, S., Hermosilla, P., Vaškevičius, N., Colosi, M., & Ropinski, T. (2024) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv62453.2024.00076 · Cited by 20
“The recent advances of large vision-language models such as CLIP [44], ALIGN [25] and follow-up works [30, 31, 66, 68] have made a paradigm shift in scene understanding from images enabling open-vocabulary object classification, detection and semantic segmentation [14, 16, 19, 29, 34].”
MethodVLDadaptor: Domain Adaptive Object Detection With Vision-Language Model Distillation
Ke, J., He, L., Han, B., Li, J., Wang, D., & Gao, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3453061 · Cited by 21
Text-Guided Explorable Image Super-Resolution
Gandikota, K. V., & Chandramouli, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02447 · Cited by 20
“Recent developments in image generation [18, 21, 57], contrastive learning[7] and large-scale training on massive internet-scale datasets of paired text prompts and images [70, 71] accelerated research in vision-language learning [30, 58, 61, 63, 64, 68].”
BackgroundLong-Tailed Anomaly Detection with Learnable Class Names
Ho, C.-H., Peng, K., & Vasconcelos, N. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01182 · Cited by 21
“LTAD overcomes this challenge by leveraging the understanding of the “abnormal” concept by the pretrained ALIGN [33] foundation model and a learned semantic descriptor for the classes in the training set.”
MethodInfluentialLearning Object State Changes in Videos: An Open-World Perspective
Xue, Z., Kumar, A., & Grauman, K. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01750 · Cited by 25
“We pioneer the use of textual state descriptions to generate a long tail of object state pseudo-labels, by leveraging the remarkable capabilities of VLMs [23, 44, 63] and large language models (LLMs) [40, 53, 54].”
MethodVisual Alignment Pre-training for Sign Language Translation
Jiao, P., Min, Y., & Chen, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_20 · Cited by 17
Robust Domain Generalization for Multi-modal Object Recognition
Qiao, Y., Li, K., Lin, J., Rong, W., Jiang, C., Luo, Y., & Yang, H. (2024) · International Conference on Artificial Intelligence and Electromechanical Automation (AIEA) · doi:10.1109/aiea62095.2024.10692575 · Cited by 29
“Diverging from conventional methods that rely on learning from images and encoded labels, contemporary progress in vision-language pre-training aims to harness naturally occurring supervision derived from extensive visual-language pairs [2], [4], [18], [19].”
BackgroundMethodTang, L., Tian, Z., Li, K., He, C., Zhou, H., Zhao, H., Li, X., & Jia, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72764-1_20 · Cited by 23
Zermatten, V., Castillo-Navarro, J., Marcos, D., & Tuia, D. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.01.006 · Cited by 22
Prefix Conditioning Unifies Language and Label Supervision
Saito, K., Sohn, K., Zhang, X., Li, C., Lee, C.-Y., Saenko, K., & Pfister, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00280 · Cited by 19
Abstract Visual Reasoning with Tangram Shapes
Ji, A., Kojima, N., Rush, N., Suhr, A., Vong, W. K., Hawkins, R., & Artzi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.38 · Cited by 39
Prefix Conditioning Unifies Language and Label Supervision
Saito, K., Sohn, K., Zhang, X., Li, C.-L., Lee, C.-Y., Saenko, K., & Pfister, T. (2022) · arXiv · doi:10.48550/arxiv.2206.01125 · Cited by 17
Zhao, G., Li, G., Chen, W., & Yu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01542 · Cited by 20
“With the development of powerful language encoders [22] and contrastive image-text training [18, 37, 47], recent works transfer the language capabilities of these models to OVD [13, 20, 29, 46, 48, 49].”
BackgroundImproving Medical Multi-modal Contrastive Learning with Expert Annotations
Kumar, Y., & Marttinen, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72661-3_27 · Cited by 29
Hierarchical and complementary experts transformer with momentum invariance for image-text retrieval
Zhang, Y., Ji, Z., Pang, Y., & Han, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112912 · Cited by 16
Towards highly sensitive deep learning-based end-to-end database search for tandem mass spectrometry
Yu, Y., & Li, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00960-1 · Cited by 19
“Most importantly, the weak-supervision regime under these frameworks requires no annotations beyond cross-modal data pairs, demonstrating increased tolerance to biases and enhanced robustness across datasets 23,24 .”
Multiple Information Prompt Learning for Cloth-Changing Person Re-Identification
Wei, S., Gao, Z., Ma, C., Zhao, Y., Guan, W., & Chen, S. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3531217 · Cited by 19
“However, the emergence of Vision-language Models (VLMs) [21], [37] provides new ideas and methods for the joint handling of cross-modal information, which enables the joint understanding and manipulation of visual and language information.”
Enhancing generalization in camera trap image recognition: Fine-tuning visual language models
Yang, Z., Tian, Y., Wang, L., & Zhang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129826 · Cited by 18
DeCLIP: Decoding CLIP Representations for Deepfake Localization
Smeu, S., Oneaţă, E., & Oneaţă, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00025 · Cited by 24
“Learning transferable representations from unlabelled data has seen impressive progress in recent years [9, 23, 46, 49].”
PERCEIVER-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Tang, Z., Cho, J., Lei, J., & Bansal, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00439 · Cited by 17
Fusing Pre-Trained Language Models with Multimodal Prompts through Reinforcement Learning
Yu, Y., Chung, J., Yun, H., Hessel, J., Park, J. S., Lu, X., Zellers, R., Ammanabrolu, P., Bras, R. J. L., Kim, G., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01044 · Cited by 17
Toward General Cross-Modal Signal Reconstruction for Robotic Teleoperation
Chen, Y., Li, A., Wu, D., & Zhou, L. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3312944 · Cited by 16
Self-citationText encoders bottleneck compositionality in contrastive vision-language models
Kamath, A., Hessel, J., & Chang, K.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.301 · Cited by 35
Distilling DETR with Visual-Linguistic Knowledge for Open-Vocabulary Object Detection
Li, L., Miao, J., Shi, D., Tan, W., Ren, Y., Yang, Y., & Pu, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00598 · Cited by 20
Self-citationHelping Hands: An Object-Aware Ego-Centric Video Recognition Model
Zhang, C., Gupta, A., & Zisserman, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01278 · Cited by 20
Space-time Prompting for Video Class-incremental Learning
Pei, Y., Qing, Z., Zhang, S., Wang, X., Zhang, Y., Zhao, D., & Qian, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01096 · Cited by 18
ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data
Varma, M., Delbrouck, J.-B., Hooper, S., Chaudhari, A. S., & Langlotz, C. P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02031 · Cited by 17
Hierarchical matching and reasoning for multi-query image retrieval
Ji, Z., Li, Z., Zhang, Y., Wang, H., Pang, Y., & Li, X. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106200 · Cited by 16
Self-citationZhu, L., Yin, W., Yang, Y., Wu, F., Zeng, Z., Gu, Q., Wang, X., Zhou, C., & Ye, N. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02036-4 · Cited by 16
Self-citationCAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-Shot Learning
Zheng, Z., Zhu, H., & Nevatia, R. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00174 · Cited by 16
Zhao, D., Lu, J., & Yuan, B. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3386934 · Cited by 18
Yang, Z., An, G., Zheng, Z., Cao, S., & Ruan, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3390133 · Cited by 16
“ALIGN [25] used a dual-encoder architecture and contrastive loss to align visual and linguistic representations of image and text pairs, which is trained on a noisy dataset of over a billion image-substitute text pairs.”
BackgroundMethodZero-Shot Temporal Action Detection by Learning Multimodal Prompts and Text-Enhanced Actionness
Raza, A., Yang, B., & Zou, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3414275 · Cited by 19
“Studies on image-text bonds include paired documents [29] and joint image-text embedding with category annotations [30], [31].”
Ump: Unified Modality-Aware Prompt Tuning for Text-Video Retrieval
Zhang, H., Zeng, P., Gao, L., Song, J., & Shen, H. T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3429192 · Cited by 18
“Recently, pre-trained vision-language models have revealed remarkable learning capabilities, which can be briefly classified into two categories: pre-trained image-text models ( e.g. , CLIP [14], ALIGN [37]) and pre-trained video-text models ( e.g. , Frozen [38], MIL-NCE [39]).”
Generative Region-Language Pretraining for Open-Ended Object Detection
Lin, C., Jiang, Y., Qu, L., Yuan, Z., & Cai, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01324 · Cited by 23
“Learning visual models from language supervision becomes popular in image recognition tasks [19, 20, 40, 41, 59, 60, 63, 65].”
BackgroundImproved Visual Grounding through Self-Consistent Explanations
He, R., Cascante-Bonilla, P., Yang, Z., Berg, A. C., & Ordóñez, V. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01244 · Cited by 20
“Vision-and-language models that are trained to associate images with text have shown to be effective for many tasks and benchmarks [21, 27, 31, 41], including object detection [18, 54] and image segmentation [15, 36, 50].”
BackgroundChen, Y., Chen, D., Liu, R., Zhou, S., Xue, W., & Peng, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01768 · Cited by 16
“In the past few years, the surge of large-scale visual-language pre-trained (VLP) models [26, 31, 34, 49, 57, 77] have revolutionized multiple fields of computer vision, including image classification, captioning [74], grounding [33], image-text retrieval, and so on.”
Self-citationCalibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
You, C., Mint, Y., Dai, W., Sekhon, J. S., Staib, L. H., & Duncan, J. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02470 · Cited by 19
Situational Awareness Matters in 3D Vision Language Reasoning
Man, Y., Gui, L.-Y., & Wang, Y.-X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01298 · Cited by 19
“Text-image contrastive models [29, 53] propose to align the feature space of two modalities with large-scale pretraining, fueling numerous downstream tasks from generalized open-vocabulary visual perception [21, 33, 35] to text-to-image generation [55].”
BackgroundMulti-view visual semantic embedding for cross-modal image–text retrieval
Li, Z., Guo, C., Wang, X., Zhang, H., & Hu, L. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111088 · Cited by 16
Self-citationVamos: Versatile Action Models for Video Understanding
Wang, S., Zhao, Q., Do, M. Q., Agarwal, N., Lee, K., & Sun, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73254-6_9 · Cited by 34
Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification
Lan, L., Wang, F., Zheng, X., Wang, Z., & Liu, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3509721 · Cited by 20
“These models, trained on vast datasets of images and text, have achieved performance levels comparable with fully supervised techniques [17], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31].”
SAM-Net: Semantic-assisted multimodal network for action recognition in RGB-D videos
Liu, D., Meng, F., Mi, J., Ye, M., Li, Q., & Zhang, J. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111725 · Cited by 18
Radiological Reports Improve Pre-training for Localized Imaging Tasks on Chest X-Rays
Müller, P., Kaissis, G. A., Zou, C., & Rueckert, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_62 · Cited by 17
Language-Guided Face Animation by Recurrent StyleGAN-Based Generator
Hang, T., Yang, H., Liu, B., Fu, J., Geng, X., & Guo, B. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3248143 · Cited by 15
Injecting Text Clues for Improving Anomalous Event Detection From Weakly Labeled Videos
Liu, T., Lam, K., & Bao, B. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3477351 · Cited by 15
“Moreover, inspired by the excellent capacity of recent vision-language (V-L) models [19], [20] (e.g., CLIP [21]) in learning powerful generic representations and aligning cross-modal semantics, it is promising to improve VAD performance by associating event texts with video sequences through…”
Background
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
