The DOI matches its DataCite record; 71 of the 75 references it cites were found in a registry or an index.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citation
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Items 801–900 of 1,011
Cacophony: An Improved Contrastive Audio-Text Model
Zhu, G., Darefsky, J., & Duan, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485170 · Cited by 27
“For the contrastive objective [14], [42], we use linear layers to map the text and audio emeddingds to the same dimension, followed by l 2 -normalization layer.”
MethodBiswas, D., & Tešić, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127857 · Cited by 18
Wen, Z., Liu, J., Zhang, H., & Zuo, F. (2025) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2025.3567126 · Cited by 16
“Recently, VLMs [23], [24] pretrained on image-text pairs using contrastive learning have shown strong generalization capabilities on downstream tasks.”
Toward general text-guided multimodal brain MRI synthesis for diagnosis and medical image analysis
Wang, Y., Xiong, H., Sun, K., Bai, S., Dai, L., Ding, Z., Liu, J., Wang, Q., Liu, Q., & Shen, D. G. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102182 · Cited by 25
“Additionally, some models 24–27 adopt contrastive language-image pre-training (CLIP) strategies 28,29 to utilize aligned text and image embed-dings to tackle downstream tasks.”
Joint Learning of Localized Representations from Medical Images and Reports
Müller, P., Kaissis, G. A., Zou, C., & Rückert, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_39 · Cited by 30
Li, J., Tang, S., Zhu, L., Zhang, W., Yang, Y., Chua, T., Wu, F. Q., & Zhuang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3274139 · Cited by 15
Self-citationBose, P., Rana, P., & Ghosh, P. (2023) · IEEE Access · doi:10.1109/access.2023.3299877 · Cited by 18
MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning
Yang, B., Liu, F., Wu, X., Wang, Y., Sun, X., & Zou, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.664 · Cited by 19
ConStruct-VL: Data-Free Continual Structured VL Concepts Learning*
Smith, J. S., Cascante-Bonilla, P., Arbelle, A., Kim, D., Panda, R., Cox, D., Yang, D., Kira, Z., Feris, R., & Karlinsky, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01440 · Cited by 15
Zero-Shot Nuclei Detection via Visual-Language Pre-trained Models
Wu, Y., Zhou, Y., Saiyin, J., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43987-2_67 · Cited by 15
Visual Grounding With Joint Multimodal Representation and Interaction
Zhu, H. B., Lu, Q., Xue, L., Xue, M., Yuan, G., & Zhong, B. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3324362 · Cited by 16
Zhang, X., Zeng, Y., Zhang, J., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.40 · Cited by 21
Enhancing Visual Grounding in Vision-Language Pre-Training With Position-Guided Text Prompts
Wang, J., Zhou, P., Shou, M. Z., & Yan, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3343736 · Cited by 17
Defense-Prefix for Preventing Typographic Attacks on CLIP
Azuma, H., & Matsui, Y. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00392 · Cited by 22
Unified Visual Relationship Detection with Vision and Language Models
Zhao, L., Yuan, L., Gong, B., Cui, Y., Schroff, F., Yang, M., Adam, H., & Liu, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00641 · Cited by 21
Vision-and-Language Navigation via Latent Semantic Alignment Learning
Wu, S., Fu, X., Wu, F. H., & Zha, Z.-J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3358112 · Cited by 19
Semantic-Oriented Visual Prompt Learning for Diabetic Retinopathy Grading on Fundus Images
Zhang, Y., Ma, X., Huang, K., Li, M., & Heng, P. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3383827 · Cited by 16
“Furthermore, prompt learning was applied to vision-language models (VLMs) to learn task-specific visual representations based on texture prompts [11], [12].”
BackgroundMethodEnhancing Visual Continual Learning with Language-Guided Supervision
Ni, B., Zhao, H., Zhang, C., Hu, K., Meng, G., Zhang, Z., & Xiang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02272 · Cited by 17
“For example, contrastive language-image pretraining demonstrates impressive “zero-shot” transfer and generalization capacities [20, 39, 55].”
Meta-prompting for Automating Zero-Shot Visual Recognition with LLMs
Mirza, M. J., Karlinsky, L., Lin, W. R., Doveh, S., Micorek, J., Koziński, M., Kuehne, H., & Possegger, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72627-9_21 · Cited by 19
Liu, G., Li, P., Zhao, Z., He, J., He, G., & Zhong, S. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104748 · Cited by 18
When Large Vision-Language Models Meet Person Re-Identification
Wang, Q., Li, B., & Xue, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460593 · Cited by 26
“Recently, visual language pre-training significantly improves the performance of many downstream tasks by training to match images and language [7, 17].”
Prompt-Ladder: Memory-efficient prompt tuning for vision-language models on edge devices
Cai, S., Liu, X., Yuan, J., & Zhou, Q. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111460 · Cited by 15
CPL: Counterfactual Prompt Learning for Vision and Language Models
He, X., Yang, D., Feng, W., Fu, T.-J., Akula, A., Jampani, V., Narayana, P., Basu, S., Wang, W. Y., & Wang, X. E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.224 · Cited by 32
Can Language Models Understand Physical Concepts?
Li, L., Xu, J., Dong, Q., Zheng, C., Sun, X., Kong, L., & Liu, Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.726 · Cited by 22
No One Representation to Rule Them All: Overlapping Features of Training MethodsPreprint
Gontijo-Lopes, R., Dauphin, Y., & Cubuk, E. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12899 · Cited by 15
Active Prompt Learning in Vision Language Models
Bang, J., Ahn, S., & Lee, J.-G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02550 · Cited by 19
“In a similar vein, ALIGN [18] was introduced, employing a comparable training methodology but featuring distinct architectural and training dataset characteristics.”
MethodReferSAM: Unleashing Segment Anything Model for Referring Image Segmentation
Liu, S.-A., Xie, H., Ge, J., & Zhang, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3524543 · Cited by 15
“Two-tower frameworks like CLIP [7] and ALIGN [23] use contrastive learning to align vision and language embeddings, achieving strong zero-shot performance [24] and generalization capabilities [25].”
Elsharif, W., Alzubaidi, M. S., & Agus, M. (2025) · IEEE Access · doi:10.1109/access.2025.3585745 · Cited by 26
What Do You MEME? Generating Explanations for Visual Semantic Role Labelling in Memes
Sharma, S. K., Agarwal, S., Suresh, T., Nakov, P., Akhtar, M. S., & Chakraborty, T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26166 · Cited by 20
Zhan, B., Li, M., Luo, W., et al. (2023) · Biology · doi:10.3390/biology12071017 · Cited by 14
From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping
Wang, J., Yan, M., Zhang, Y., & Sang, J. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/481 · Cited by 23
ViLEM: Visual-Language Error Modeling for Image-Text Retrieval
Chen, Y., Ma, Z., Zhang, Z., Qi, Z., Yuan, C., Shan, Y., Li, B., Hu, W., Qie, X., & Wu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01060 · Cited by 15
Self-citationLANIT: Language-Driven Image-to-Image Translation for Unlabeled Data
Park, J., Kim, S., Kim, S.-H., Cho, S., Yoo, J., Uh, Y., & Kim, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02241 · Cited by 18
Zara, G., Conti, A., Roy, S., Lathuilière, S., Rota, P., & Ricci, E. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00946 · Cited by 16
EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
Chen, J., Guo, L., Sun, J., Shao, S., Yuan, Z., Lin, L., & Zhang, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27872 · Cited by 18
Fine-Granularity Alignment for Text-Based Person Retrieval Via Semantics-Centric Visual Division
Wei, Z., Zhang, Z. Y., Wu, P., Wang, J., Wang, P., & Zhang, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3392831 · Cited by 15
“ALIGN [49] is similar to CLIP in the learning scheme, but it demonstrates the effectiveness of training on a large-scale dataset with noisy text supervision.”
MethodMiao, Q., Wu, X., Xu, C., Ji, Y., Zuo, W., Guo, Y., & Meng, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111860 · Cited by 15
Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
Shukor, M., Thome, N., & Cord, M. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104071 · Cited by 15
An artificial intelligence approach for interpreting creative combinational designs
Chen, L., Xiao, S., Chen, Y., Sun, L., Childs, P., & Han, J. (2024) · Journal of Engineering Design · doi:10.1080/09544828.2024.2377068 · Cited by 22
“For instance, CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021), and Imagebert (Qi et al. 2020) all effectively perform joint representation of text and images.”
Self-citationHierarchy-Aware Interactive Prompt Learning for Few-Shot Classification
Yin, X., Wu, J., Yang, W., Xu, Z., Zhang, S., & Zhang, T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3432753 · Cited by 17
Zhang, L., Awal, R., & Agrawal, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01307 · Cited by 25
Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
Wang, J., & Kang, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00393 · Cited by 14
Edge-Aware 3D Instance Segmentation Network with Intelligent Semantic Prior
Roh, W., Jung, H., Nam, G., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01951 · Cited by 14
“Recently, visual language models [17, 32] have highlighted the benefits of their capability to provide general embedded knowledge from large-scale image-text multimodal data.”
BackgroundLeveraging Cross-Modal Neighbor Representation for Improved CLIP Classification
Yi, C., Ren, L., Zhan, D., & Ye, H.-J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02587 · Cited by 19
InVERGe: Intelligent Visual Encoder for Bridging Modalities in Report Generation
Deria, A., Kumar, K., Chakraborty, S., Mahapatra, D., & Roy, S. S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00208 · Cited by 17
“These architectures incorporate the dual-encoder architecture [24, 39], the fusion-encoder architecture [34], the encoder-decoder architecture [14, 51] and more recently, the unified transformer architecture - Bootstrap-ping Language-Image Pre-training (BLIP) [30] and Beit [49].”
MarineInst: A Foundation Model for Marine Image Analysis with Instance Visual Description
Zheng, Z., Chen, Y., Zeng, H., Vu, T.-A., Hua, B., & Yeung, S.-K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72627-9_14 · Cited by 18
Self-citationFaceptor: A Generalist Model for Face Perception
Qin, L., Wang, M., Liu, X., Zhang, Y., Deng, W., Song, X., Xu, W., & Deng, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72754-2_14 · Cited by 14
A Review on Vision-Language-Based Approaches: Challenges and Applications
Pham, M.-T., Pham, Q.-H., Tran, Q.-D., Ho, H.-T., Nguyen, L. V., Huy, D. N. M., & Nguyen, T. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.060363 · Cited by 19
FindIt: Generalized Localization with Natural Language Queries
Kuo, W., Bertsch, F., Li, W., Piergiovanni, A., Saffar, M. H., & Angelova, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_29 · Cited by 14
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation
Zhu, Y., Sun, Z., Cheng, S., Huang, L., Wu, L., & Wang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.168 · Cited by 14
Distilling Vision-Language Models on Millions of Videos
Zhao, Y., Zhao, L., Zhou, X., Wu, J., Chu, C.-T., Miao, H., Schroff, F., Adam, H., Liu, T., Gong, B., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01245 · Cited by 23
“We attribute the alt-text’s inferior performance to a compounding error of textual noise [25], spurious correlation when computing visual similarities [71], and the visual discrepancy between images and videos.”
BackgroundLet Segment Anything Help Image DehazePreprint
Jin, Z., Chen, S., Chen, Y., Xu, Z., & Feng, H. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4611106 · Cited by 14
Self-citationPixel-level semantic parsing in complex industrial scenarios using large vision-language models
Ji, X., Gong, F., Wang, N., Zhao, Y., Ma, Y., & Shi, Z. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102794 · Cited by 13
LLM-infused bi-level semantic enhancement for corporate credit risk prediction
Lu, S., Su, Y., Zhang, X., Chai, J., & Yu, L. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104091 · Cited by 15
Malagoli, E., & Persio, L. D. (2025) · Mathematics · doi:10.3390/math13060893 · Cited by 19
Truong, T.-D., Nguyen, H.-Q., Nguyen, X.-B., Dowling, A. P. G., Li, X., & Luu, K. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02521-4 · Cited by 17
A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues
Li, Y., Hu, B., Chen, X., Ding, Y., Ma, L., & Zhang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.601 · Cited by 20
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
Tan, R., Ray, A., Burns, A., Plummer, B. A., Salamon, J., Nieto, O., Russell, B. C., & Saenko, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01019 · Cited by 13
When are Lemons Purple? The Concept Association Bias of Vision-Language Models
Tang, Y., Yamada, Y., Zhang, Y., & Yildirim, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.886 · Cited by 48
Fine-Tuning for Few-Shot Image Classification by Multimodal Prototype Regularization
Wu, Q., Qi, J., Zhang, D., Zhang, H., & Tang, J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3379896 · Cited by 17
Compound Text-Guided Prompt Tuning via Image-Adaptive Cues
Tan, H., Li, J., Zhou, Y., Wan, J., Lei, Z., & Zhang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28311 · Cited by 15
CAD - Contextual Multi-modal Alignment for Dynamic AVQA
Nadeem, A., Hilton, A., Dawes, R., Thomas, G., & Mustafa, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00709 · Cited by 16
“Unlike the alignment between co-occurring modalities in AV tasks, which has an inherent nature, transformer-based alignment techniques primarily leverage large amounts of data for vision-language tasks [40, 51, 64, 75, 107, 109].”
BackgroundŞahin, U., Li, H., Khan, Q., Cremers, D., & Volker Tresp, C. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00547 · Cited by 17
“…data crawled from the internet [39, 40], the unsupervised contrastive learning paradigm stands out as a primary approach to pre-train VLMs [18, 34, 47] [36], hard negative mining based on learned embeddings became a popular approach to improve contrastive learning [36], with different…”
Gambetti, A., & Han, Q. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31437 · Cited by 20
Review of Zero-Shot Remote Sensing Image Scene Classification
Tan, X., Xi, B., Li, J., Zheng, T., Li, Y., Xue, C., & Chanussot, J. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3410995 · Cited by 14
Zhu, L., Wang, Y., Hu, Y., Su, X., & Fu, K. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3417421 · Cited by 14
Unlocking Robotic Autonomy: A Survey on the Applications of Foundation Models
Jang, D.-S., Cho, D.-H., Lee, W.-C., Ryu, S.-K., Jeong, B.-M., Hong, M., Jung, M., Kim, M., Lee, M., Lee, S., et al. (2024) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0438-7 · Cited by 17
Label Propagation for Zero-shot Classification with Vision-Language Models
Stojnić, V., Kalantidis, Y., & Tolias, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02190 · Cited by 14
VIT-LENS: Towards Omni-modal Representations
Lei, W., Ge, Y., Yi, K., Zhang, J., Gao, D., Sun, D., Ge, Y., Shan, Y., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02516 · Cited by 21
“L ENS that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained-ViT and aligning them to a pre-defined space.”
BackgroundScaling Laws for Data Filtering—Data Curation Cannot be Compute Agnostic
Goyal, S., Maini, P., Lipton, Z. C., Raghunathan, A., & Kolter, J. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02142 · Cited by 44
“Past works on scaling laws [25, 26] estimate the error of a model (at a given parameter count) after training for n samples as: y = an b + d , where a, d > 0 and b 0 are constants to be determined empirically, and y is a performance metric such as the loss of the model on a validation set.”
BackgroundSyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
Song, C. H., Hwang, T., Yoon, J., Choi, S., & Gu, Y. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01323 · Cited by 14
“Recently, there has been rapid progress in developing Vision-Language Pretraining (VLP) [32, 38, 6, 39, 19, 18, 42, 23, 36, 20, 27, 3], paving the way to bridge the gap between visual and textual features.”
BackgroundMethodOST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition
Chen, T., Yu, H., Yang, Z., Li, Z., Sun, W., & Chen, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01787 · Cited by 15
“Large-scale contrastive language-image pre-training [25, 46, 65] have shown remarkable performance in various computer vision tasks.”
BackgroundSelf-citationOpen-Set Recognition in the Age of Vision-Language Models
Miller, D., Sünderhauf, N., Kenna, A., & Mason, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_1 · Cited by 17
Adapting Vision-Language Models via Learning to Inject Knowledge
Xuan, S., Yang, M., & Zhang, S. S. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3468884 · Cited by 15
“For instance, CLIP [1] and ALIGN [16] align image and text features into a common embedding space through contrastive learning, achieving impressive zero-shot accuracy.”
MethodBie, Y., Luo, L., Chen, Z., & Chen, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_72 · Cited by 23
Leveraging Temporal Contextualization for Video Action Recognition
Kim, M., Han, D., Kim, T., & Han, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72664-4_5 · Cited by 13
OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation
Wang, Z., Li, Y., Liu, T., Zhao, H., & Wang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72970-6_5 · Cited by 18
Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation
Zhang, Z., Ke, W., Zhu, Y., Liang, X., Liu, J., Ye, Q., & Zhang, T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3504816 · Cited by 15
“The other stream focuses on transferring the alignment capability of image-text pretrained models [12, 30, 41, 50, 60] to the pixel-text level during fine-tuning stage.”
Knowledge-Enhanced Visual-Language Pretraining for Computational Pathology
Zhou, X., Zhang, X., Wu, C., Zhang, Y., Xie, W., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72943-0_20 · Cited by 20
Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation
Fang, H., Wu, P., Li, Y., Zhang, X., & Lu, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72897-6_13 · Cited by 14
Consistent prompt learning for vision-language models
Zhang, Y., & Tian, X. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112974 · Cited by 15
Liu, Y., Liu, L., Zheng, Y., Liu, Y., Dang, F., Li, N., & Ma, K. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4303-8 · Cited by 16
“VLMs such as CLIP [200], ALIGN [201], and FLAVA [202] have enhanced visual semantic understanding by integrating visual and language features.”
Zero-Shot Detection of Visual Food Safety Hazards via Knowledge-Enhanced Feature Synthesis
Guo, L., Hu, X., Liu, W., & Liu, Y. (2025) · Applied Sciences · doi:10.3390/app15116338 · Cited by 13
Relaxing Contrastiveness in Multimodal Representation Learning
Lin, Z., Bas, E., Singh, K. Y., Gurumurthy, S., & Bhotika, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00226 · Cited by 14
FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning
Mirchandani, S., Yu, L., Wang, M., Sinha, A., Jiang, W., Xiang, T., & Zhang, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.716 · Cited by 13
Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning
Liu, Y., Chen, C., Wang, C., King, X., & Liu, M. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612106 · Cited by 14
Singh, H., Zhang, P., Wang, Q., Wang, M., Xiong, W., Du, J., & Chen, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.56 · Cited by 22
Self-citationPrompting Scientific Names for Zero-Shot Species Recognition
Parashar, S., Lin, Z., Li, Y., & Kong, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.610 · Cited by 16
Gouidis, F., Papantoniou, K., Papoutsakis, K. E., Patkos, T., Argyros, A., & Plexousakis, D. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31190 · Cited by 13
“Therefore, in addition to OaSc, we have chosen to employ additionally three state-of-the-art Vision Language Models (VLMs) that support this functionality: CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021), and BLIP (Li et al. 2022).”
MethodSat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images
Dhakal, A., Ahmad, A., Khanal, S., Sastry, S., Kerner, H., & Jacobs, N. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00058 · Cited by 13
Prioritized Semantic Learning for Zero-Shot Instance Navigation
Sun, X., Liu, L., Zhi, H., Qiu, R., & Liang, J. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73254-6_10 · Cited by 12
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 346
Adaptive CLIP for open-domain 3D model retrieval
Song, D., Qiang, Z., Zhang, C., Wang, L., Liu, Q., Yang, Y., & Liu, A.-A. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103989 · Cited by 12
Self-citationEntity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval
Dong, X. M., Zhan, X., Wei, Y., Wei, X.-Y., Wang, Y., Lu, M., Cao, X., & Liang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3291237 · Cited by 14
CLIPText: A New Paradigm for Zero-shot Text Classification
Qin, L., Wang, W., Chen, Q., & Che, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.69 · Cited by 14
MAGVLT: Masked Generative Vision-and-Language Transformer
Kim, S., Jo, D., Lee, D., & Kim, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02235 · Cited by 12
Response generation in multi-modal dialogues with split pre-generation and cross-modal contrasting
Li, L., Zhang, D., Zhu, S., Li, S., & Zhou, G. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103581 · Cited by 12
MVP-SEG: Multi-view Prompt Learning for Open-Vocabulary Semantic Segmentation
Guo, J., Wang, Q., Gao, Y., Jiang, X., Lin, S., & Zhang, B. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8555-5_13 · Cited by 12
Structures Aware Fine-Grained Contrastive Adversarial Hashing for Cross-Media Retrieval
Liang, M., Li, Y., Yu, Y., Cao, X., Xue, Z., Li, A., & Lu, K. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3356258 · Cited by 14
“CLIP [47] and ALIGN [48] are the recent proposed multi-modal pre-training models based on comparative learning, which form a task-agnostic model by predicting which text matches which image.”
BackgroundMulti-Task Learning of Object States and State-Modifying Actions From Web Videos
Souček, T., Alayrac, J.-B., Miech, A., Laptev, I., & Šivic, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3362288 · Cited by 17
Metadata
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
