The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 401–500 of 1,011
Token-Mixer: Bind Image and Text in One Embedding Space for Medical Image Reporting
Yang, Y. Q., Yu, J., Fu, Z., Zhang, K., Yu, T., Wang, X., Jiang, H., Lv, J., Huang, Q., & Han, W. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3412402 · Cited by 45
“For instance, CLIP [11] and ALIGN [12] collect large collections of image-text pairs and train models using contrastive learning by predicting whether the images and texts are matched.”
BackgroundSelf-citationCoralSCOP: Segment any COral Image on this Planet
Zheng, Z., Liang, H., Hua, B., Wong, Y. H., Ang, P., Chui, A. P. Y., & Yeung, S.-K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02661 · Cited by 44
“Foundation Models (such as CLIP [46], ALIGN [27] and SAM [31]) have been widely favored by the whole CV community.”
CLIP-DINOiser: Teaching CLIP a Few DINO Tricks for Open-Vocabulary Semantic Segmentation
Wysoczańska, M., Siméoni, O., Ramamonjisoa, M., Bursuc, A., Trzcinski, T. P., & Pérez, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73030-6_18 · Cited by 72
Text-Image Alignment for Diffusion-Based Perception
Kondapaneni, N., Marks, M., Knott, M., Guimarães, R., & Perona, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01317 · Cited by 55
BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning
Liang, S., Zhu, M., Liu, A., Wu, B., Cao, X., & Chang, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02327 · Cited by 84
“Another line of MCL such as Unicoder-VL [18], Uniter [8], and ALIGN [15] employed the random sampling of negative samples from either images or texts to enable the model to determine their match.”
BackgroundFER-Former: Multimodal Transformer for Facial Expression Recognition
Li, Y., Wang, M., Gong, M., Lu, Y., & Liu, L. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521788 · Cited by 44
SATR: Zero-Shot Semantic Segmentation of 3D Shapes
Abdelreheem, A., Skorokhodov, I., Ovsjanikov, M., & Wonka, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01392 · Cited by 48
Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training
Luo, D., Huang, J., Gong, S., Jin, H., & Liu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02207 · Cited by 42
A Simple Framework for Text-Supervised Semantic Segmentation
Yi, M., Cui, Q., Wu, H., Yang, C., Yoshie, O., & Lu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00683 · Cited by 44
Chen, Y., Yuan, J., Tian, Y., Geng, S., Li, X., Zhou, D., Metaxas, D., & Yang, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01449 · Cited by 46
Self-citationDisentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning
Qing, Z., Zhang, S., Huang, Z., Zhang, Y., Gao, C., Zhao, D., & Sang, N. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01281 · Cited by 41
Li, F., Jiang, Q., Zhang, H., Ren, T., Liu, S., Zou, X., Xu, H., Li, H., Yang, J., Li, C., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01222 · Cited by 54
“Innovations in open-vocabulary object detection [7, 12, 19,24,26,43,44,47] and open-vocabulary segmentation [6, 8, 14, 31, 37, 44], have shown great potential in generic visual perception, by leveraging large pre-trained vision-language models like CLIP [30] and ALIGN [10].”
MethodOne Prompt Word is Enough to Boost Adversarial Robustness for Pre-Trained Vision-Language Models
Li, L., Guan, H., Qiu, J., & Spratling, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02304 · Cited by 42
“Large pre-trained Vision-Language Models (VLMs) such as CLIP [51], ALIGN [27], BLIP [33], etc . have emerged as general-purpose (a.k.a. foundation) models [5], fostering ecosystems across numerous sectors within the realm of artificial intelligence [5, 50].”
Deep learning based infrared small object segmentation: Challenges and future directions
Yang, Z., Yu, H., Zhang, J., Tang, Q., & Mian, A. S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103007 · Cited by 41
Playing Lottery Tickets with Vision and Language
Gan, Z., Chen, Y.-C., Li, L., Chen, T., Cheng, Y., Wang, S., Liu, J., Wang, L., & Liu, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19945 · Cited by 48
Self-citationLAFITE: Towards Language-Free Training for Text-to-Image Generation
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T. (2021) · arXiv · doi:10.48550/arxiv.2111.13792 · Cited by 38
Hallucination of Multimodal Large Language Models: A Survey
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024) · arXiv · doi:10.48550/arxiv.2404.18930 · Cited by 38
“Among them, vision+LLM is particularly prominent, owing to the extensive research on vision-language models (VLMs) [51, 88, 116] prior to LLMs.”
Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning
Guo, X., Selvaraj, N. M., Yu, Z., Kong, A. W., Shen, B., & Kot, A. C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02023 · Cited by 44
Lin, W., Karlinsky, L., Shvetsova, N., Possegger, H., Koziński, M., Panda, R., Feris, R., Kuehne, H., & Bischof, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00267 · Cited by 45
Collaborating Foundation Models for Domain Generalized Semantic Segmentation
Benigmim, Y., Roy, S., Essid, S., Kalogeiton, V. S., & Lathuilière, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00300 · Cited by 48
“The FMs comprise of contrastively trained image classification models (e.g., CLIP [56], ALIGN [30]), text-conditioned generative models (e.g., Stable Diffusion [59], DALL-E [57]), vision transformer-based segmentation model trained on mammoth dataset (e.g., Segment Anything Model (SAM) [35]), to…”
MethodImproved Zero-Shot Classification by Adapting VLMs with Text Descriptions
Saha, O., Horn, G. V., & Maji, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01661 · Cited by 51
“Zero-shot Image Classification using VLMs. Vision Language models (VLMs) [6, 11, 29, 33, 45] learn to associate images with their corresponding text captions.”
BackgroundMethodA multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Liu, F., Zheng, L., Yin, Q., Huang, J., Luo, J., Thakur, A., Branson, K., Schwab, P., Yin, B., Wu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01339-7 · Cited by 50
CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning
Bansal, H., Yin, F., Singhi, N., Grover, A., Yu, Y., & Chang, K.-W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00017 · Cited by 59
HGPrompt: Bridging Homogeneous and Heterogeneous Graphs for Few-Shot Prompt Learning
Yu, X., Fang, Y., Liu, Z., & Zhang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29596 · Cited by 42
Aligning and Prompting Everything All at Once for Universal Visual Perception
Shen, Y., Fu, C., Chen, P., Zhang, M., Li, K., Sun, X. W., Wu, Y., Lin, S., & Ji, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01253 · Cited by 45
“The first one is to learn all-purposed visual features via self-supervised learning, e.g ., DINO [1] and iBOT [2], and align text-image corpora with weakly-supervised learning, such as CLIP [3] and ALIGN [4].”
BackgroundCLIB-FIQA: Face Image Quality Assessment with Confidence Calibration
Ou, F.-Z., Li, C., Wang, S., & Kwong, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00167 · Cited by 36
CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
Sun, S., Li, R., Torr, P. H. S., Gu, X., & Li, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01251 · Cited by 52
“Integration with Grad-CAM.”
InfluentialTransformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval
Shukor, M., Couairon, G., Grechka, A., & Cord, M. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00503 · Cited by 36
CapDet: Unifying Dense Captioning and Open-World Detection Pretraining
Long, Y., Wen, Y., Han, J., Xu, H., Ren, P., Zhang, W., Zhao, S., & Liang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01462 · Cited by 39
Integrating Language Guidance Into Image-Text Matching for Correcting False Negatives
Li, Z., Guo, C., Feng, Z., Hwang, J., & Du, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3261443 · Cited by 38
Self-citationPosition-Guided Text Prompt for Vision-Language Pre-Training
Wang, J., Zhou, P., Shou, M. Z., & Yan, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02226 · Cited by 39
CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval
Pei, R., Liu, J., Li, W., Shao, B., Xu, S., Dai, P., Lu, J., & Yan, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01820 · Cited by 43
Change Detection With Cross-Domain Remote Sensing Images: A Systematic Review
Chen, J., Hou, D., He, C., et al. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3416183 · Cited by 35
LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP
Huang, Y., Shakeri, F., Dolz, J., Boudiaf, M., Bahig, H., & Ayed, I. B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02244 · Cited by 49
“3.2). scale vision-language models (VLMs), such as CLIP [23] and ALIGN [15], have attracted wide attention and made substantial progress in computer vision, showing promising generalization capabilities in various downstream tasks.”
BackgroundInfluentialLearning Transferable Negative Prompts for Out-of-Distribution Detection
Li, T., Pang, G., Bai, X. P., Miao, W., & Zheng, J. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01665 · Cited by 50
“Concurrently, other researchers [1, 24, 49] have also shown the remarkable generalization capabilities of CLIP and similar vision-language models [17] in various downstream tasks [7, 20, 56].”
Multimodal Fake News Detection with Contrastive Learning and Optimal Transport
Shen, X., Huang, M., Hu, Z. F., Cai, S., & Zhou, T. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1473457 · Cited by 47
Webly Supervised Concept Expansion for General Purpose Vision Models
Kamath, A., Clark, C., Gupta, T., Kolve, E., Hoiem, D., & Kembhavi, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_38 · Cited by 40
CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning
Cheng, Y., Wei, F., Bao, J., Chen, D., & Zhang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01823 · Cited by 43
Multi-modal multi-task feature fusion for RGBT tracking
Cai, Y., Sui, X., & Gu, G. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101816 · Cited by 34
Efficient Image Captioning for Edge Devices
Wang, N., Xie, J., Luo, H., Cheng, Q., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25359 · Cited by 39
ResFormer: Scaling ViTs with Multi-Resolution Training
Tian, R., Wu, Z., Dai, Q., Hu, H., Qiao, Y., & Jiang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02176 · Cited by 42
DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition With Limited Annotations
Hu, P., Sun, X., Sclaroff, S., & Saenko, K. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3346405 · Cited by 42
Adapt and explore: Multimodal mixup for representation learning
Lin, R., & Hu, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102216 · Cited by 34
Learning Concise and Descriptive Attributes for Visual Recognition
Yan, A., Wang, Y., Zhong, Y., Dong, C., He, Z., Lu, Y., Wang, W. Y., Shang, J., & McAuley, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00287 · Cited by 43
Gabeff, V., Rußwurm, M., Tuia, D., & Mathis, A. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02026-6 · Cited by 40
“The loss aims at maximizing the similarity of the positive pairs and minimizing it for negative pairs: Here, the likelihoods following Equations (3 and 4), where τ is the temperature parameter: At inference time, CLIP is used to compute the cosine similarity between queries and images.”
Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation
Yamazaki, K., Hanyu, T., Vo, K., Pham, T. M., Tran, M., Doretto, G., Nguyen, A., & Le, N. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610193 · Cited by 52
AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning
Huang, D., Xiong, X., Ma, J., Li, J., Jie, Z., Ma, L., & Li, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00309 · Cited by 34
“Leveraging large-scale datasets such as [38, 39], vision foundation models have made significant advancements in various tasks of computer vision, encompassing image classification [14, 35], segmentation [16, 49, 68], and generation [37].”
BackgroundKnowledge-Enhanced Dual-Stream Zero-Shot Composed Image Retrieval
Suo, Y., Ma, F., Zhu, L., & Yang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02545 · Cited by 45
“Vision-language pre-training has been a long-standing research topic with wide real-world applications [13, 18, 27, 33, 53, 54, 81].”
BackgroundSelf-citationTowards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture
Gupta, T., Kamath, A., Kembhavi, A., & Hoiem, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01591 · Cited by 65
LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval
Luo, Z., Zhao, P., Xu, C., Geng, X., Shen, T., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01029 · Cited by 43
Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., & Qi, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3410324 · Cited by 34
A Review of Transformer-Based Approaches for Image Captioning
Ondeng, O., Absaloms, H. O., & Akuon, P. O. (2023) · Applied Sciences · doi:10.3390/app131911103 · Cited by 39
Going Beyond Nouns With Vision & Language Models Using Synthetic Data
Cascante-Bonilla, P., Shehada, K., Smith, J. S., Doveh, S., Kim, D., Panda, R., Varol, G., Oliva, A., Ordóñez, V., Feris, R., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01844 · Cited by 41
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Mensink, T., Uijlings, J., Castrejón, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araújo, A. D. M., & Ferrari, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00289 · Cited by 61
Bayesian Prompt Learning for Image-Language Model Generalization
Derakhshani, M. M., Sanchez, E., Bulat, A., da Costa, V. G. T., Snoek, C. G. M., Tzimiropoulos, G., & Martínez, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01398 · Cited by 61
Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
Li, J., Gao, M., Wei, L., Tang, S., Zhang, W., Li, M., Ji, W., Tian, Q., Chua, T., & Zhuang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00241 · Cited by 37
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
ControlRoom3D: Room Generation Using Semantic Proxy Rooms
Schult, J., Tsai, S. S., Höllein, L., Wu, B., Wang, J., Ma, C., Li, K., Wang, X., Wimbauer, F., He, Z., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00593 · Cited by 58
“Fueled by the availability of large-scale vision-text datasets [22, 32, 43–45, 52, 57] and powerful vision-language models [35], recent advancements in 2D diffusion models [5, 11, 31, 36, 40, 41] have significantly contributed to the ongoing progress in generating 3D objects and scenes.”
BackgroundGeneralizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction
Li, H., Chen, Y., Chen, Y., Yu, R., Yang, W., Wang, L., Ding, B., & Han, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01083 · Cited by 34
“Recently, Vision-Language Models (VLMs) [11, 25, 33] underscored the significance of integrating multimodal information for developing robust encoders.”
BackgroundSelf-citationGalLoP: Learning Global and Local Prompts for Vision-Language Models
Lafon, M., Ramzi, E., Rambour, C., Audebert, N., & Thome, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73030-6_15 · Cited by 42
Adam, M., Albaseer, A. M., Baroudi, U. A., & Abdallah, M. M. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3554537 · Cited by 36
i-Code: An Integrative and Composable Multimodal Learning Framework
Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Yi‐chong, X., Qian, Y., Gao, M., Chen, Y., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26290 · Cited by 45
Self-citationCLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose
Zhang, X., Wang, W., Chen, Z., Xu, Y., Zhang, J., & Tao, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02229 · Cited by 36
MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models
Ko, D., Choi, J., Choi, H. K., On, K.-W., Roh, B., & Kim, H. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01925 · Cited by 38
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
Dai, W., Liu, Z., Ji, Z., Su, D., & Fung, P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.156 · Cited by 59
Knowledge Distillation with Segment Anything (SAM) Model for Planetary Geological Mapping
Julka, S., & Granitzer, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53969-5_6 · Cited by 33
Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective
Wu, W., Sun, Z., Song, Y., Wang, J., & Ouyang, W. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01876-w · Cited by 36
How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
Ming, Y., & Li, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01895-7 · Cited by 39
MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning
Shu, C., Zhu, Y., Tang, X., Xiao, J., Chen, Y., Li, X., Zhang, Q., & Lu, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121526 · Cited by 34
Self-citationGeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
Li, Z., Zhou, W., Chiang, Y., & Chen, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.317 · Cited by 49
Self-citationYang, Y., Huang, W., Wei, Y., Peng, H., Jiang, X., Jiang, H., Wei, F., Yin, W., Han, H., Qiu, L., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00260 · Cited by 32
Self-citationFollow Anything: Open-Set Detection, Tracking, and Following in Real-Time
Maalouf, A., Jadhav, N., Murthy, J. K., Chahine, M., Vogt, D. M., Wood, R. J., Torralba, A. B., & Rus, D. L. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3366013 · Cited by 38
Luo, S. X., Chen, W., Tian, W., Liu, R., Hou, L., Zhang, X., Shen, H., Wu, R., Geng, S., Zhou, Y., et al. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3406372 · Cited by 40
“With the popularity of multi-modal learning, researchers have also explored its potential for aligning image-text representation [10], [18], [78], [136], [140], [149].”
BackgroundSelf-citationLeSAM: Adapt Segment Anything Model for Medical Lesion Segmentation
Gu, Y., Wu, Q., Tang, H. N., Mai, X., Shu, H., Li, B., & Chen, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3406871 · Cited by 37
Self-citationPrompt-Driven Referring Image Segmentation with Instance Contrasting
Shang, C., Song, Z., Qiu, H., Wang, L., Meng, F., & Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00395 · Cited by 37
“Recently, with the success of applying vision-language pre-trained models e.g. , CLIP [38] and ALIGN [17], to downstream tasks, prompt learning has been further extended to computer vision [1, 18] and multi-modal learning [25, 45, 52, 53] tasks.”
BackgroundMethodTowards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching
Chu, M., Zheng, Z., Ji, W., Wang, T., & Chua, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73247-8_13 · Cited by 46
“Current datasets typically provide pairs of images, focusing on matching images from disparate platforms like drones and satellites [28,32,51,64,75,83].”
BackgroundLanguage-Guided Hierarchical Fine-Grained Image Forgery Detection and Localization
Guo, X. B., Liu, X., Masi, I., & Liu, X. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02255-9 · Cited by 38
“Secondly, unlike the prior work that assumes a “flat” structure between different attributes, we represent all forgery attributes in a hierarchical way, exploring dependencies among them.”
Instruction-Guided Scene Text Recognition
Du, Y., Chen, Z., Su, Y., Jia, C., & Jiang, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3525526 · Cited by 41
“The condition means knowledgegiveninadvance,andthe question - answer paircorre-sponds to the specified query and its response.”
Self-citationA Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., & Plummer, B. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_18 · Cited by 49
Jiang, Y., Zhang, L., & Wang, W. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.220 · Cited by 44
Dual Alignment Unsupervised Domain Adaptation for Video-Text Retrieval
Hao, X., Zhang, W., Wu, D., Zhu, F., & Li, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01818 · Cited by 32
Open-Set Fine-Grained Retrieval via Prompting Vision-Language Evaluator
Wang, S., Chang, J., Li, H., Wang, Z., Ouyang, W., & Tian, Q. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01857 · Cited by 32
Black Box Few-Shot Adaptation for Vision-Language models
Ouali, Y., Bulat, A., Matinez, B., & Tzimiropoulos, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01424 · Cited by 39
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Bugliarello, E., Liu, F., Pfeiffer, J., Reddy, S., Elliott, D., Ponti, E. M., & Vulić, I. (2022) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.2201.11732 · Cited by 39
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 54
Shen, C., Cai, G., Tian, J., Wu, X., Ding, M., Wang, B., & Liu, D. (2023) · Food Chemistry · doi:10.1016/j.foodchem.2023.138265 · Cited by 31
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
Chen, Z., Diao, S., Wang, B., Li, G., & Wan, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02139 · Cited by 34
Pre-Trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
Wang, S., Zhang, J., Yuan, Z., & Shan, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02313 · Cited by 41
“Vision-language models pre-trained on large dataset have achieved impressive success in numerous tasks, such as image classification [22, 36], text-to-image generation [37, 38], image caption [7, 27].”
BackgroundLearning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection
Li, J., Zhang, J., Li, J., Li, G., Liu, S., Lin, L., & Li, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01578 · Cited by 31
“…vision tasks [11, 12, 18–22, 39 Leveraging large-scale pre-trained Vision and Language Models (PVLMs), exemplified by CLIP [29] and ALIGN [13], recent advances in OVD, e.g. [4 , 7, 35], employ knowledge distillation to transfer insightful knowledge of PVLMs to the task of object…”
MethodSegment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation Without Manual Labels
Huang, R., Peng, S., Takmaz, A., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72754-2_16 · Cited by 31
Fang, Z., Ren, J., Zheng, J., Chen, R., & Zhao, H. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3519173 · Cited by 35
An Overview of Deep Neural Networks for Few-Shot Learning
Zhao, J., Kong, L., & Lv, J. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020049 · Cited by 33
Improving Zero-shot Generalization and Robustness of Multi-Modal Models
Ge, Y., Ren, J., Gallagher, A., Wang, Y., Yang, M., Adam, H., Itti, L., Lakshminarayanan, B., & Zhao, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01067 · Cited by 37
Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World
Yu, Q., Li, J., Wu, Y., Tang, S., Ji, W., & Zhuang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01971 · Cited by 42
I can’t believe there’s no images! : Learning Visual Tasks Using Only Language Supervision
Gu, S., Clark, C., & Kembhavi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00252 · Cited by 43
DeAR: Debiasing Vision-Language Models with Additive Residuals
Seth, A., Hemani, M., & Agarwal, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00659 · Cited by 34
GeneCIS: A Benchmark for General Conditional Image Similarity
Vaze, S., Carion, N., & Misra, I. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00663 · Cited by 35
Mapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 31
ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation
Hu, X., Zhang, K., Xia, L., Chen, A. Y., Luo, J., Sun, Y., Wang, K., Qiao, N., Zeng, X., Sun, M., et al. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00297 · Cited by 41
Global–Local Information Soft-Alignment for Cross-Modal Remote-Sensing Image–Text Retrieval
Hu, G. Y., Wen, Z., Lv, Y., Zhang, J., & Wu, Q. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3401031 · Cited by 30
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
