Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Open accessAbstract
Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
VQ-VAE-2 improves upon traditional VAE methods and excels in multimodal generation, generating high-quality images or other modality content.
“VQ-VAE-2 [189] improves upon traditional VAE methods and excels in multimodal generation tasks, capable of generating high-quality images or other modality content.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [189]
The Introduction (p. 1) and Conclusion (p. 9) describe ALIGN, an image–text representation model used for retrieval, not VQ-VAE-2. Searches for VQ-VAE-2 and variational-autoencoder-based multimodal generation found no passage addressing the claimed improvement or generation capabilities. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5× · see citing works
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al.”
BackgroundMethodInfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1× · see citing works
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1× · see citing works
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1× · see citing works
“Previously word2vec (Mikolov et al., 2013a;b) shows that linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5× · see citing works
“, 2018), CLIP (Radford et al., 2021), BiT (Kolesnikov et al.”
BackgroundMethodResultInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1× · see citing works
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundSelf-citationVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 1× · see citing works
“However, vision-language pre-training datasets such as Conceptual Captions (Sharma et al., 2018), Visual Genome Dense Captions (Krishna et al., 2016), and ImageBERT (Qi et al., 2020) require even heavier work on human annotation, semantic parsing, cleaning and balancing.”
BackgroundDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 1× · see citing works
“Recently, self-supervised (Chen et al., 2020b; Tian et al., 2020; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al., 2019; Xie et al., 2020; Pham et al., 2020) have been studied as alternative paradigms.”
Background3D Object Representations for Fine-Grained Categorization
Krause, J., Stark, M. M., Deng, J., & Fei-Fei, L. (2013) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.77 · Cited by 3,525 · cited 1× · see citing works
“, 2012), Stanford Cars (Krause et al., 2013), and Food101 (Bossard et al.”
BackgroundAutomated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · see citing works
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 2× · see citing works
“Recently, self-supervised (Hénaff et al., 2019; Chen et al., 2020b; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al.”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · cited 2× · see citing works
“In the vision domain, pre-training on large-scale supervised data such as ImageNet (Deng et al., 2009), OpenImages (Kuznetsova et al., 2020), and JFT-300M (Sun et al., 2017; Kolesnikov et al., 2020) has proven to be critical for improving performance on downstream tasks via transfer learning.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 1× · see citing works
“The train/dev/test splits are defined in Young et al. (2014).”
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393 · cited 4× · see citing works
“, 2021), NoisyStudent (Xie et al., 2020), and Meta-Pseudo-Labels (Pham et al.”
BackgroundInfluentialSelf-citationKuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00982 · Cited by 2,194 · see citing works
Self-citationViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
Food-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Gool, L. V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,105 · see citing works
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · cited 7× · see citing works
“This is one of the most effective loss functions for both self-supervised (Chen et al., 2020b) and supervised (Zhai & Wu, 2019; Musgrave et al., 2020) representation learning.”
BackgroundMethodInfluentialSelf-citationTian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · cited 2× · see citing works
“Recently, self-supervised (Chen et al., 2020b; Tian et al., 2020; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al.”
BackgroundConceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · cited 1× · see citing works
“Pre-training has also become the de-facto approach in vision-language modeling (Lu et al., 2019; Chen et al., 2020c; Li et al., 2020).”
BackgroundMicrosoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · cited 12× · see citing works
“Image-text retrieval results on Flickr30K and MSCOCO datasets (zero-shot and fine-tuned).”
MethodInfluentialParkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,564 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · cited 1× · see citing works
“Recently, self-supervised (Chen et al., 2020b; Tian et al., 2020; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al., 2019; Xie et al., 2020; Pham et al., 2020) have been studied as alternative paradigms.”
BackgroundUnifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsPreprint
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,386 · see citing works
Learning Fine-Grained Image Similarity with Deep Ranking
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J. F., Chen, B., & Wu, Y. N. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.180 · Cited by 1,282 · see citing works
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250 · cited 2× · see citing works
“, 2009) and its variants including ImageNet-R(endition) (Hendrycks et al., 2020) (non-natural images such as art, cartoons, sketches), ImageNet-A(dversarial) (Hendrycks et al.”
BackgroundExploring the Limits of Weakly Supervised Pretraining
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., & van der Maaten, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_12 · Cited by 1,224 · see citing works
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
Hill, F., Reichart, R., & Korhonen, A. (2015) · Computational Linguistics · doi:10.1162/coli_a_00237 · Cited by 1,179 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T. (2024) · TIB Data Manager · doi:10.57702/iuuvgtaz · Cited by 1,140 · see citing works
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090 · cited 1× · see citing works
“…ILSVRC-2012 benchmark (Deng et al., 2009) and its variants including ImageNet-R(endition) (Hendrycks et al., 2020) (non-natural images such as art, cartoons, sketches), ImageNet-A(dversarial) (Hendrycks et al., 2021) (more challenging images for ML models), and ImageNet-V2 (Recht et al., 2019).”
BackgroundBig Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_29 · Cited by 1,039 · cited 11× · see citing works
“Table 6 reports the mean accuracy (including the breakdown results on each subgroup) with standard deviation from three fine-tuning runs and shows that ALIGN outperforms BiT-L (Kolesnikov et al., 2020) with similar hyper-parameter selection method applied.”
BackgroundMethodInfluentialData-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014 · cited 1× · see citing works
“Recently, self-supervised (Hénaff et al., 2019; Chen et al., 2020b; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al.”
BackgroundDeep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 802 · cited 2× · see citing works
“…et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been proposed.”
BackgroundMethodDo ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790 · cited 3× · see citing works
“We first apply zero-shot transfer of ALIGN to visual classifi-cation tasks on ImageNet ILSVRC-2012 benchmark (Deng et al., 2009) and its variants including ImageNet-R(endition) (Hendrycks et al., 2020) (non-natural images such as art, cartoons, sketches), ImageNet-A(dversarial) (Hendrycks et al., 2021) (more challenging…”
BackgroundGrounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · cited 2× · see citing works
“…(Frome et al., 2013; Faghri et al., 2018) and improved versions (e.g., leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been…”
BackgroundMethodSelf-citationDual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.232 · Cited by 749 · cited 2× · see citing works
“, leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been proposed.”
BackgroundVisual Semantic Reasoning for Image-Text Matching
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00475 · Cited by 628 · cited 5× · see citing works
“R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Zero-shot (Faghri et al., 2018), VSRN (Li et al., 2019), DE I2T (Parekh et al., 2021), and DE T2T+I2T (Parekh et al., 2021). image R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 VSE++ ERNIE-ViL (Yu et al., 2020), VILLA (Gan et al., 2020) and Oscar (Li et al., 20…”
BackgroundMethodInfluentialLarge Batch Optimization for Deep Learning: Training BERT in 76 minutesPreprint
You, Y., Li, J., Reddi, S. J., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J. W., Keutzer, K., & Hsieh, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00962 · Cited by 597 · cited 2× · see citing works
“We use LAMB optimizer (You et al., 2020)1 with weight decay ratio 1e-5.”
MethodContrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · see citing works
(2021) · International Conference on Learning Representations · Cited by 449 · see citing works
A Metric Learning Reality Check
Musgrave, K., Belongie, S., & Lim, S.-N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_41 · Cited by 447 · cited 2× · see citing works
“This is one of the most effective loss functions for both self-supervised (Chen et al., 2020b) and supervised (Zhai & Wu, 2019; Musgrave et al., 2020) representation learning.”
BackgroundBillion-scale semi-supervised learning for image classificationPreprint
Yalnız, İ. Z., Jeǵou, H., Chen, K., Paluri, M., & Mahajan, D. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00546 · Cited by 387 · see citing works
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · cited 2× · see citing works
“These works are however limited to small datasets such as Flickr (Joulin et al., 2015; Li et al., 2017) and COCO Captions (Desai & Johnson, 2020; Sariyildiz et al., 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.”
BackgroundPixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal TransformersPreprint
Huang, Z., Zeng, Z., Liu, B., Fu, D., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00849 · Cited by 344 · cited 2× · see citing works
“We evaluate the zero-shot model performance of ALIGN and compare it with M 3 P (Huang et al., 2020a) and UC2 (Zhou et al., 2021).”
BackgroundMethodLarge-Scale Adversarial Training for Vision-and-Language Representation LearningPreprint
Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06195 · Cited by 343 · see citing works
Self-citationLearning Visual Features from Large Weakly Supervised Data
Joulin, A., van der Maaten, L., Jabri, A., & Vasilache, N. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_5 · Cited by 315 · cited 3× · see citing works
“These works are however limited to small datasets such as Flickr (Joulin et al., 2015; Li et al., 2017) and COCO Captions (Desai & Johnson, 2020; Sariyildiz et al., 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.”
BackgroundLearning the Best Pooling Strategy for Visual Semantic Embedding
Chen, J., Hu, H., Wu, H., Jiang, Y., & Wang, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01553 · Cited by 314 · cited 7× · see citing works
“This is one of the most effective loss functions for both self-supervised (Chen et al., 2020b) and supervised (Zhai & Wu, 2019; Musgrave et al., 2020) representation learning.”
BackgroundMethodInfluentialA Large-scale Study of Representation Learning with the Visual Task Adaptation BenchmarkPreprint
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lučić, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04867 · Cited by 249 · see citing works
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text DataPreprint
Qi, D., Su, L., Song, J. L., Cui, E., Bharti, T., & Sacheti, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07966 · Cited by 201 · cited 6× · see citing works
“With fine-tuning, ALIGN outperforms all existing methods by a large margin, including those that employ more complex cross-modal attention layers such as ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020c), Table 1.”
BackgroundMethodInfluentialClassification is a Strong Baseline for Deep Metric LearningPreprint
Zhai, A., & Wu, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12649 · Cited by 196 · see citing works
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives.
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2018) · Cited by 184 · see citing works
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders
Messina, N., Amato, G. M., Esuli, A., Falchi, F., Gennaro, C., & Marchand‐Maillet, S. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3451390 · Cited by 158 · cited 2× · see citing works
“, leveraging object detectors, dense feature maps, or multi-attention layers) (Socher et al., 2014; Karpathy et al., 2014; Kiros et al.; Nam et al., 2017; Li et al., 2019; Messina et al., 2020; Chen et al., 2020a) have been proposed.”
BackgroundFindings of the Third Shared Task on Multimodal Machine Translation
Barrault, L., Bougares, F., Specia, L., Lala, C., Elliott, D., & Frank, S. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6402 · Cited by 142 · cited 1× · see citing works
“We test the multilingual model on Multi30k, a multilingual image text retrieval dataset extends Flickr30K (Plum-mer et al., 2015) to German (de) (Elliott et al., 2016), French (fr) (Elliott et al., 2017) and Czech (cs) (Barrault et al., 2018).”
MethodLearning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 120 · cited 3× · see citing works
“These works are however limited to small datasets such as Flickr (Joulin et al., 2015; Li et al., 2017) and COCO Captions (Desai & Johnson, 2020; Sariyildiz et al., 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.”
BackgroundERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene GraphPreprint
Yu, F. R., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16934 · Cited by 106 · cited 4× · see citing works
“, 2020c), ERNIE-ViL (Yu et al., 2020), VILLA (Gan et al.”
MethodInfluentialLearning Visual N-Grams from Web Data
Li, A., Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.449 · Cited by 91 · cited 12× · see citing works
“Image-text retrieval results on Flickr30K and MSCOCO datasets (zero-shot and fine-tuned).”
BackgroundInfluentialAligning Visual Regions and Textual Concepts for Semantic-Grounded Image RepresentationsPreprint
Liu, F., Liu, Y., Ren, X., He, X., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06139 · Cited by 63 · cited 2× · see citing works
“This is at least an order of magnitude smaller than their counterparts in the vision domain, and much smaller than large corpora of text from the internet for NLP pre-training (e.g., Devlin et al. (2019); Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b); Raffel et al. (2020)).”
BackgroundIllustrative Language Understanding: Large-Scale Visual Grounding with Image Search
Kiros, J., Chan, W., & Hinton, G. E. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1085 · Cited by 49 · see citing works
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
Parekh, Z., Baldridge, J., Cer, D. M., Waters, A., & Yang, Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.249 · Cited by 49 · cited 7× · see citing works
“We also evaluate ALIGN on Crisscrossed Captions (CxC) (Parekh et al., 2021), which is an extension of MSCOCO with additional human semantic similarity judgments for caption-caption, image-image, and image-caption pairs.”
BackgroundMethodInfluentialSelf-citationPrototypical contrastive learning of unsupervised representationsPreprint
Li, J. (2024) · arXiv (Cornell University) · doi:10.57702/qpm0hks1 · Cited by 46 · see citing works
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-trainingPreprint
Ni, M., Huang, H., Su, L., Cui, E., Bharti, T., Wang, L., Gao, J., Zhang, D., & Duan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02635 · Cited by 40 · see citing works
Graph-RISE: Graph-Regularized Image Semantic EmbeddingPreprint
Juan, D.-C., Lu, C.-T., Li, Z., Peng, F., Timofeev, A., Chen, Y., Gao, Y., Duerig, T., Tomkins, A., & Ravi, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10814 · Cited by 29 · cited 2× · see citing works
“High-quality visual representations for classification or retrieval are usually pre-trained on large-scale labeled datasets (Mahajan et al., 2018; Kolesnikov et al., 2020; Dosovitskiy et al., 2021; Juan et al., 2020).”
BackgroundSelf-citationMeta Pseudo LabelsPreprint
Pham, H., Dai, Z., Xie, Q., et al. (2020) · arXiv (Cornell University) · Cited by 19 · see citing works
Self-citationUC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingPreprint
Zhou, M., Zhou, L., Wang, S., Yu, C., Li, L., Zhou, Y., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00332 · Cited by 7 · cited 2× · see citing works
“We evaluate the zero-shot model performance of ALIGN and compare it with M 3 P (Huang et al., 2020a) and UC2 (Zhou et al., 2021).”
Method
Flagged first, then by citations
4 of the 75 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language , 2016.
- Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers , September 2017.
- Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Proceedings of Neural Information Processing Systems , 2013.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems , 2019.
- Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., and Le, Q. V. Rethinking pre-training and selftraining. In Advances in Neural Information Processing Systems , 2020.
Cited by
Sorted by citations
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · see citing works
“The target datasets are ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNetR (Hendrycks et al., 2021a), all of which have compatible class names with ImageNet allowing seamless transfer for the prompts learned by CoOp.”
BackgroundConditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,918 · see citing works
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs
Ding, X., Zhang, X., Han, J., & Ding, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01166 · Cited by 1,511 · see citing works
Sigmoid Loss for Language Image Pre-Training
Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01100 · Cited by 1,127 · see citing works
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,341 · see citing works
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232 · see citing works
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,197 · see citing works
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · see citing works
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01832 · Cited by 1,011 · see citing works
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966 · see citing works
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76 · see citing works
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · see citing works
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
“For our image and text pairs we first leverage the ALIGN [50] dataset, composed of 1.”
BackgroundMethodInfluentialYOLO-World: Real-Time Open-Vocabulary Object Detection
Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., & Shan, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01599 · Cited by 815 · see citing works
“Recent works [9, 14, 48, 53, 59] have explored the prevalent vision-language models [19, 39] to address open-20”
BackgroundMedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.256 · Cited by 838 · see citing works
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734 · see citing works
“Vision models, trained on pure-vision data or aligned with the BERT series [52, 54, 70], often exhibit representation inconsistencies with LLMs. (3) Inefficient connection.”
BackgroundMethodObjaverse: A Universe of Annotated 3D Objects
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsanit, K., Kembhavi, A., & Farhadi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01263 · Cited by 725 · see citing works
Reproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · see citing works
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
RegionCLIP: Region-based Language-Image Pretraining
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01629 · Cited by 623 · see citing works
WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation
Jeong, J., Zou, Y., Kim, T., Zhang, D., Ravichandran, A., & Dabeer, O. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01878 · Cited by 585 · see citing works
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J. S., Tian, J., Zhang, Y. S., Shi, Z., Jianping Fan, B., & He, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 614 · see citing works
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 597 · see citing works
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561 · see citing works
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01855 · Cited by 540 · see citing works
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01398 · Cited by 536 · see citing works
Segment anything model for medical images?
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103061 · Cited by 536 · see citing works
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Extract Free Dense Labels from CLIP
Zhou, C., Loy, C. C., & Dai, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_40 · Cited by 513 · see citing works
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 522 · see citing works
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing
Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., & Zhou, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390838 · Cited by 507 · see citing works
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
Jain, A., Tancik, M., & Abbeel, P. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00583 · Cited by 477 · see citing works
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., & Marculescu, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00682 · Cited by 484 · see citing works
GroupViT: Semantic Segmentation Emerges from Text Supervision
Xu, J., Mello, S. D., Liu, S., Byeon, W., Breuel, T. M., Kautz, J., & Wang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01760 · Cited by 475 · see citing works
Xia, W., Zhang, Y., Yang, Y., Xue, J., Zhou, B., & Yang, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3181070 · Cited by 496 · see citing works
Self-citationTinyViT: Fast Pretraining Distillation for Small Vision Transformers
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19803-8_5 · Cited by 480 · see citing works
Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification
Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_29 · Cited by 464 · see citing works
Open-vocabulary Object Detection via Vision and Language Knowledge DistillationPreprint
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13921 · Cited by 581 · see citing works
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · see citing works
“Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
MethodPrompting Visual-Language Models for Efficient Video Understanding
Ju, C., Han, T., Zheng, K., Zhang, Y., & Xie, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_7 · Cited by 423 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · see citing works
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · arXiv · doi:10.48550/arxiv.2104.14548 · Cited by 405 · see citing works
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404 · see citing works
Chen, K., Liu, C., Chen, H., Zhang, H., Li, W., Zou, Z., & Shi, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3356074 · Cited by 416 · see citing works
Zero-Shot Text-Guided Object Generation with Dream Fields
Jain, A., Mildenhall, B., Barron, J. T., Abbeel, P., & Poole, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00094 · Cited by 395 · see citing works
OpenScene: 3D Scene Understanding with Open Vocabularies
Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., & Funkhouser, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00085 · Cited by 395 · see citing works
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H. R., Hovy, E. H., & Sun, Y. (2022) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 404 · see citing works
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402 · see citing works
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., & Mello, S. D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00289 · Cited by 382 · see citing works
Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., & Li, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01369 · Cited by 368 · see citing works
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01763 · Cited by 366 · see citing works
Simple Open-Vocabulary Object Detection
Minderer, M., Gritsenko, A. A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_42 · Cited by 365 · see citing works
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
Ghiasi, G., Gu, X., Cui, Y., & Lin, T.-Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_31 · Cited by 357 · see citing works
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., & Ling, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19772-7_1 · Cited by 361 · see citing works
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · see citing works
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Jiang, D., & Ye, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00273 · Cited by 348 · see citing works
Segment anything model for medical image segmentation: Current applications and future directions
Zhang, Y., Shen, Z., & Jiao, R. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108238 · Cited by 348 · see citing works
CLIP-Driven Fine-Grained Text-Image Person Re-Identification
Yan, S., Dong, N., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3327924 · Cited by 347 · see citing works
Side Adapter Network for Open-Vocabulary Semantic Segmentation
Xu, M., Zhang, Z., Wei, F., Hu, H., & Bai, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00288 · Cited by 335 · see citing works
CLIP-ReID: Exploiting Vision-Language Model for Image Re-identification without Concrete Text Labels
Li, S., Li, S., & Li, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25225 · Cited by 335 · see citing works
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Metzer, G., Richardson, E., Patashnik, O., Giryes, R., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01218 · Cited by 336 · see citing works
Prompt-aligned Gradient for Prompt Tuning
Zhu, B., Niu, Y., Han, Y., Wu, Y., & Zhang, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01435 · Cited by 328 · see citing works
Decoupling Zero-Shot Semantic Segmentation
Ding, J., Xue, N., Xia, G.-S., & Dai, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01129 · Cited by 313 · see citing works
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373 · see citing works
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.05482 · Cited by 307 · see citing works
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304 · see citing works
“To avoid these steps, Jia et al. in ALIGN [43] gather one billion noisy image-caption pairs from Conceptual Captions Dataset [54].”
InfluentialVision-Language Pre-Training with Triple Contrastive Learning
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., & Huang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01522 · Cited by 309 · see citing works
FILIP: Fine-grained Interactive Language-Image Pre-TrainingPreprint
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 388 · see citing works
Self-citationSLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 285 · see citing works
VLP: A Survey on Vision-language Pre-training
Chen, F., Zhang, D., Han, M., Chen, X.-Y., Shi, J. C., Xu, S., & Xu, B. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1369-5 · Cited by 295 · see citing works
Towards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 296 · see citing works
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization
Yao, H., Zhang, R., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00653 · Cited by 273 · see citing works
RMT: Retentive Networks Meet Vision Transformers
Fan, Q., Huang, H., Chen, M., Liu, H., & He, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00539 · Cited by 278 · see citing works
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S. A., Yang, M., & Khan, F. S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01394 · Cited by 274 · see citing works
Lu, Y., Liu, J., Zhang, Y., Liu, Y., & Tian, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00514 · Cited by 262 · see citing works
How Segment Anything Model (Sam) Boost Medical Image Segmentation: A SurveyPreprint
Zhang, Y., & Jiao, R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4495221 · Cited by 263 · see citing works
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
Yang, A., Nagrani, A., Seo, P. H., Miech, A., Pont-Tuset, J., Laptev, I., Šivic, J., & Schmid, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01032 · Cited by 266 · see citing works
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02240 · Cited by 255 · see citing works
ShareGPT4V: Improving Large Multi-modal Models with Better Captions
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., & Lin, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72643-9_22 · Cited by 337 · see citing works
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation
Zhou, Z., Lei, Y., Zhang, B., Liu, L., & Liu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01075 · Cited by 251 · see citing works
Transformer-Based Visual Segmentation: A Survey
Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3434373 · Cited by 272 · see citing works
MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation
Chen, C., Miao, J., Wu, D., Zhong, A., Yan, Z., Kim, S., Hu, J. J., Liu, Z., Sun, L., Li, X., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103310 · Cited by 246 · see citing works
X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval
Gorti, S. K., Vouitsis, N., Ma, J., Golestan, K., Volkovs, M., Garg, A., & Yu, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00495 · Cited by 239 · see citing works
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 241 · see citing works
Vision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 257 · see citing works
“A prevalent strategy involves intensive pre-training on datasets where images/videos are paired with their corresponding textual descriptions.”
VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection
Wu, P., Zhou, X., Pang, G., Zhou, L., Yan, Q., Wang, P., & Zhang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28423 · Cited by 249 · see citing works
Large AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · see citing works
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey
Chen, Y., Mancini, M., Zhu, X., & Akata, Z. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3201576 · Cited by 235 · see citing works
Self-citationWhat does a platypus look like? Generating customized prompts for zero-shot image classification
Pratt, S., Covert, I., Liu, R., & Farhadi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01438 · Cited by 240 · see citing works
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01442 · Cited by 227 · see citing works
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 245 · see citing works
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., & Malekshan, K. R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01805 · Cited by 225 · see citing works
ConceptFusion: Open-set multimodal 3D mapping
Jatavallabhula, K., Kuwajerwala, A., Gu, Q., Omama, M., Iyer, G., Saryazdi, S., Chen, T., Maalouf, A., Li, S., Keetha, N., et al. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.066 · Cited by 228 · see citing works
Domain Adaptation via Prompt Learning
Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., & Huang, G. M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3327962 · Cited by 223 · see citing works
Unified Contrastive Learning in Image-Text-Label Space
Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., & Gao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01857 · Cited by 202 · see citing works
Recognize Anything: A Strong Image Tagging Model
Zhang, Y., Huang, X., Ma, J., Li, Z., Luo, Z., Xie, Y., Qin, Y., Luo, T., Li, Y., Liu, S., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00179 · Cited by 242 · see citing works
Self-citation
- Authors
- Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 4904-4916
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.05918
- OpenAlex
- W3126337491
- Semantic Scholar
- 141a5033d9994242b18bb3b217e79582f1ee9306
- Cited by
- 1,977
- References
- 75
- Citations by year
- 2026: 16 · 2025: 392 · 2024: 765 · 2023: 581 · 2022: 159 · 2021: 64
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language representation learning, contrastive learning, dual-encoder architecture, visual representation learning, zero-shot image classification, image-text retrieval, data curation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
