Learning to Prompt for Vision-Language Models
Open accessThe DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; none of the 20 sources its claims cite is checked for support yet.
Abstract
Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming—one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt’s context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.
Overview
CoChat Summary
TL;DR
CoOp automates prompt engineering for CLIP-like vision-language models by learning continuous context vectors while keeping all pre-trained parameters frozen. Across 11 datasets, it turns CLIP into a strong few-shot learner, beating hand-crafted prompts and often outperforming linear probes, while also improving robustness to some domain shifts.
Key findings
- With as few as 1–2 shots, CoOp beats hand-crafted prompts; with 16 shots the average gain is around 15%, with the highest reaching over 45% (EuroSAT >45%, DTD >20%).
- CoOp outperforms the linear probe baseline overall: the linear probe needs more than 4 shots on average to match zero-shot CLIP, while CoOp already shows an impressive average gain at 4 shots.
- On domain generalization from ImageNet, CoOp improves over zero-shot CLIP on several shifted sets, e.g. ResNet-50 source ImageNet 62.95 vs. 58.18, ImageNetV2 55.11 vs. 51.34, ImageNet-A 22.12 vs. 21.65; linear probe is much worse on target datasets.
- Unified context generally performs better, while class-specific context is preferred for some fine-grained and specialized tasks at 16 shots but mostly underperforms unified context with fewer than 8 shots.
Methods
- CoOp models a prompt’s context words as learnable vectors, with two implementations: unified context shared across classes and class-specific context; the class token can be placed at the end or in the middle, and training minimizes cross-entropy while freezing CLIP.
- Experiments use 11 image classification datasets under CLIP’s few-shot protocol with 1, 2, 4, 8, and 16 shots, full test sets, and averages over three runs; default backbone is ResNet-50, context length M=16, SGD with initial learning rate 0.002 and cosine annealing.
- Baselines are zero-shot CLIP with hand-crafted prompts and a linear probe; additional analyses cover context length, vision backbones, prompt ensembling, initialization, and domain generalization on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
Limitations
- The paper states that CoOp’s results are relatively difficult to interpret, like other continuous prompt learning methods, and that CoOp is sensitive to noisy labels, as shown by weak performance on Food101.
- The paper also notes that interpreting learned prompts via nearest vocabulary words could be inaccurate, and that there is no golden rule for selecting context length because one must balance performance and robustness to distribution shift.
- Evident from the text: class-specific context mostly underperforms unified context in low-data scenarios with fewer than 8 shots, and Table 1 shows CoOp is not uniformly better than zero-shot CLIP on every target set, e.g. ResNet-50 with M=16 on ImageNet-R is 54.96 vs. zero-shot 56.00.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 15 Not checked · 67 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that for pre-trained vision-language models, the text input, known as prompt, plays a key role in downstream datasets.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We later show that CoOp obtains much stronger performance than the linear probe model in domain generalization.”
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe that when using manual initialization (like 'a photo of a'), the nearest words for the converged vectors are mostly the ones used for initialization.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average results are shown in Figure 5(a), which indicate that having more context tokens leads to better performance and that positioning the class token in the middle gains more momentum with longer context length.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given 16 shots for training, the average gap brought by CoOp can be further increased to around 15%.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
46 more claims
Create a free account to see all 84 claims and to verify the 20 references still open.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 17
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“prompt learning methods [2,5,8,16,18,32,42] emerge recently as efficient and effective solutions for adapting pre-trained transformers to the target domain via only training very few parameters (i.e., prompts), and achieve comparable performance with finetuning the whole heavy model.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [42]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ B Results on DOSCO-2k · p. 11
“All models are trained on the training set and the checkpoints with the best validation performance are used for final test in unseen domains.”
§ References · p. 12
“Among the three approaches, CoOp and its follow-up, CoCoOp, contain learnable components while CLIP here denotes the zero-shot model.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Few-Shot Learning · p. 8
“| Zero-Shot CLIP | 61.62 | 54.81 | 38.71 | 28.05 | 64.38 |”
§ 4.2 Domain Generalization · p. 9
“| Prompt engineering | 58.18 | 61.26 | 62.05 | 66.73 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“A common approach for building state-of-the-art visual recognition systems is to train vision models to predict for a fixed set of object categories using discrete labels (He et al., 2016; Dosovitskiy et al., 2021).”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Datasets We select 11 publicly available image classification datasets used in CLIP: ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei et al.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 4×
“, 2016) or ViT (Dosovitskiy et al., 2021)—with a fixed set of weights that are seen as visual concepts and initialized randomly.”
BackgroundMethodInfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“There are some differences that distinguish our approach from the prompt learning methods developed in NLP for language models (e.g., GPT-3 (Brown et al., 2020)).”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 19×
“We briefly introduce vision-language pre-training with a particular focus on CLIP (Radford et al., 2021).”
BackgroundMethodInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 2×
“, 2017), ii) large-minibatch contrastive representation learning (Chen et al., 2020; He et al., 2020; Hénaff et al., 2020), and iii) web-scale training”
BackgroundUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · cited 4×
“For generic objects (ImageNet & Caltech101), scenes (SUN397) and actions (UCF101), using unified context is clearly better.”
BackgroundInfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“…language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“…language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
BackgroundPlaces: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · cited 1×
“Unlike existing domain generalization datasets where the domain labels are manually defined and often limited to image style variations, DOSCO-2k focuses on broader contextual domain shift, which is automatically detected by a neural network pre-trained on the Places dataset (Zhou et al., 2017).”
Background3D Object Representations for Fine-Grained Categorization
Krause, J., Stark, M., Deng, J., & Fei-Fei, L. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.77 · Cited by 3,496
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · cited 1×
“(Nilsback and Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), DTD (Cim-poi et al., 2014), EuroSAT (Helber et al., 2019) and UCF101 (Soomro et al., 2012) (see Appendix A for their statistics).”
BackgroundSUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283 · cited 3×
“For generic objects (ImageNet & Caltech101), scenes (SUN397) and actions (UCF101), using unified context is clearly better.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 2×
“Most related to our work are continuous prompt learning methods (Zhong et al., 2021; Li and Liang, 2021; Lester et al., 2021) which optimize continuous vectors in the word embedding space.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 2×
“Most related to our work are continuous prompt learning methods (Zhong et al., 2021; Li and Liang, 2021; Lester et al., 2021) which optimize continuous vectors in the word embedding space.”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 6×
“…to the adaptation of large vision-4 language models in computer vision—which we view as an important topic for democratizing foundation models (Bommasani et al., 2021)—and justify that prompt learning not only brings significant improvements to computer vision tasks in terms of transfer…”
BackgroundMethodInfluentialSelf-citationFei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 3×
“Most related to our work are continuous prompt learning methods (Zhong et al., 2021; Li and Liang, 2021; Lester et al., 2021) which optimize continuous vectors in the word embedding space.”
BackgroundDescribing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438
Fei-Fei, L., Fergus, R., & Perona, P. (2007) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2005.09.012 · Cited by 2,359 · cited 3×
“For generic objects (ImageNet & Caltech101), scenes (SUN397) and actions (UCF101), using unified context is clearly better.”
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
Food-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Van Gool, L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,072 · cited 8×
“We observe that a few words are somewhat relevant to the tasks, such as “enjoyed” for Food101, “fluffy” and “paw” for OxfordPets, and “pretty” for DTD.”
BackgroundInfluentialScaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 2×
“The target datasets are ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNetR (Hendrycks et al., 2021a), all of which have compatible class names with ImageNet allowing seamless transfer for the prompts learned by CoOp.”
BackgroundJia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
Helber, P., Bischke, B., Dengel, A., & Borth, D. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2918242 · Cited by 1,951 · cited 6×
“Tuning the sentence structure could bring further improvements, e.g., putting “a type of flower” after the class token for Flowers102, keeping only “texture” in the context for DTD, and adding “centered” before “satellite photo” for EuroSAT.”
BackgroundInfluentialLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 1×
“Knowledge probing for large pre-trained language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021;…”
BackgroundConditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,796 · cited 9×
“…pave the way for future research on efficient adaptation methods for emerging foundation models, which is still a nascent research topic. benchmark (Zhou et al., 2022b) contains 7 image recognition datasets, which cover a wide range of classification problems, such as generic object recognition,…”
BackgroundMethodInfluentialSelf-citationParkhi, O. M., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248092 · Cited by 1,533
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 2×
“…language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
BackgroundDomain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315 · cited 1×
“Since CoOp requires training on a specific data distribution, it risks learning spurious correlations that are detrimental to generalization in unseen distributions (domains), as suggested in recent studies (Taori et al., 2020; Zhou et al., 2021).”
ResultSelf-citationAutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090
Fine-Grained Visual Classification of Aircraft
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., & Vedaldi, A. (2013) · INRIA a CCSD electronic archive server · doi:10.48550/arxiv.1306.5151 · Cited by 1,084 · cited 3×
“%) on most fine-grained datasets including Flow-ers102, StanfordCars and FGVCAircraft, as well as on scene and action recognition datasets (i.e., SUN397 & UCF101).”
BackgroundHendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1907.07174 · Cited by 1,058
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014 · cited 1×
“…learning with Transformers (Vaswani et al., 2017), ii) large-minibatch contrastive representation learning (Chen et al., 2020; He et al., 2020; H´enaff et al., 2020), and iii) web-scale training datasets—CLIP benefits from 400 million curated image-text pairs while ALIGN exploits 1.8 billion…”
BackgroundInfluentialHow Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002 · cited 9×
“Though the performance is excellent, the results of CoOp are relatively difficult to interpret, like other continuous prompt learning methods in NLP.”
BackgroundMethodInfluentialZero-Shot Learning Through Cross-Modal Transfer
Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997 · cited 4×
“embedding space has been studied since nearly a decade ago (Socher et al., 2013; Frome et al., 2013; Elhoseiny et al., 2013), but with drastically different technologies.”
BackgroundMethodInfluentialNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
A simple framework for contrastive learning of visual representations
Chen, T. (2024) · TIB Data Manager · doi:10.57702/iuuvgtaz · Cited by 973
Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_16 · Cited by 952
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790 · cited 1×
“The target datasets are ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al.”
BackgroundFLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01519 · Cited by 554 · cited 4×
“diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
BackgroundInfluential(2021) · International Conference on Learning Representations · Cited by 449
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · cited 4×
“diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
BackgroundInfluentialSelf-citationVirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01101 · Cited by 342 · cited 2×
“Matching images and text features has been formulated as metric learning (Frome et al., 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al., 2017), and the recently proposed captioning (Desai and Johnson, 2021).”
BackgroundLearning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Xing, E. P., & Lipton, Z. C. (2019) · arXiv · doi:10.48550/arxiv.1905.13549 · Cited by 337 · cited 1×
“, 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al.”
Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions
Ba, J. L., Swersky, K., Fidler, S., & Salakhutdinov, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.483 · Cited by 324 · cited 1×
“For text features extraction, early work has mainly utilized pre-trained word vectors (Socher et al., 2013; Frome et al., 2013) or the hand-crafted TF-IDF features (Elhoseiny et al., 2013; Lei Ba et al., 2015).”
MethodLearning Visual Features from Large Weakly Supervised Data
Joulin, A., van der Maaten, L., Jabri, A., & Vasilache, N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_5 · Cited by 302 · cited 2×
“, 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al.”
BackgroundPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · arXiv · doi:10.48550/arxiv.2101.00190 · Cited by 289 · cited 3×
“Most related to our work are continuous prompt learning methods (Zhong et al., 2021; Li and Liang, 2021; Lester et al., 2021) which optimize continuous vectors in the word embedding space.”
BackgroundTent: Fully Test-time Adaptation by Entropy Minimization
Wang, D., Shelhamer, E., Liu, S., Olshausen, B. A., & Darrell, T. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10726 · Cited by 278
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274
Contrastive Learning of Medical Visual Representations from Paired Images and Text
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 265 · cited 4×
“…language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
BackgroundMethodInfluentialWrite a Classifier: Zero-Shot Learning Using Purely Textual Descriptions
Elhoseiny, M., Saleh, B., & Elgammal, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.321 · Cited by 257 · cited 2×
“For text features extraction, early work has mainly utilized pre-trained word vectors (Socher et al., 2013; Frome et al., 2013) or the hand-crafted TF-IDF features (Elhoseiny et al., 2013; Lei Ba et al., 2015).”
BackgroundMethodHelber, P., Bischke, B., Dengel, A., & Borth, D. (2018) · IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium · doi:10.1109/igarss.2018.8519248 · Cited by 232
Measuring Robustness to Natural Distribution Shifts in Image Classification
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., & Schmidt, L. (2020) · arXiv · doi:10.48550/arxiv.2007.00644 · Cited by 206 · cited 1×
“Since CoOp requires training on a specific data distribution, it risks learning spurious correlations that are detrimental to generalization in unseen distributions (domains), as suggested in recent studies (Taori et al., 2020; Zhou et al., 2021).”
ResultData-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J. (2020) · International Conference on Machine Learning · Cited by 168
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
Li, Y., Liang, F., Zhao, L.-C., Cui, Y., Ouyang, W., Shao, J., Yu, F., & Yan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05208 · Cited by 128 · cited 2×
“By pre-training at a large scale, models can learn diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
BackgroundMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124 · cited 2×
“…language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu et al., 2021b).”
BackgroundCLIP-Adapter: Better Vision-Language Models with Feature AdaptersPreprint
Gao, P., Geng, S., Zhang, R., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04544 · Cited by 113 · cited 1×
“It would also be interesting to investigate more generic adaptation methods for mega-size vision models (Jia et al., 2022; Bahng et al., 2022; Gao et al., 2021).”
BackgroundSelf-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces
Gomez, L., Patel, Y., Rusinol, M., Karatzas, D., & Jawahar, C. V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.218 · Cited by 106 · cited 2×
“, 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al.”
BackgroundLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A., & Riedel, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01066 · Cited by 94 · cited 1×
“Knowledge probing for large pre-trained language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021;…”
BackgroundLearning Visual N-Grams from Web Data
Li, A., Jabri, A., Joulin, A., & van der Maaten, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.449 · Cited by 84 · cited 2×
“Matching images and text features has been formulated as metric learning (Frome et al., 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al., 2017), and the recently proposed captioning (Desai and Johnson, 2021).”
BackgroundLearning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Lipton, Z. C., & Xing, E. P. (2019) · neural information processing systems · Cited by 68
Domain Generalization in Vision: A SurveyPreprint
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2021) · arXiv (Cornell University) · Cited by 65
Self-citationCLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPPreprint
Fürst, A., Rumetshofer, E., Lehner, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11316 · Cited by 38 · cited 2×
“diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
BackgroundFLAVA: A Foundational Language And Vision Alignment ModelPreprint
Singh, A., Hu, R., Goswami, V., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.04482 · Cited by 36 · cited 4×
“diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
BackgroundInfluentialSelf-supervised learning of visual features through embedding images into text topic spacesPreprint
Gómez, L., Patel, Y., Rusiñol, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08631 · Cited by 18 · cited 2×
“, 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al.”
BackgroundOn-Device Domain GeneralizationPreprint
Zhou, K., Zhang, Y., Zang, Y., Yang, J., Loy, C. C., & Liu, Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.07521 · Cited by 8
Self-citation
Research works printed but not linked: 1
- Bossard, L., Guillaumin, M., & Van Gool, L. (2014). Food-101–mining discriminative components with random forests. In ECCV · DOI · Paper page
Cited by
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,796
“[62], recent successes in visionlanguage models are mainly attributed to the developments in i) Transformers [48], ii) contrastive representation learning [4, 17, 20], and iii) web-scale training datasets [24, 40].”
BackgroundMethodInfluentialSelf-citationCLIP-Adapter: Better Vision-Language Models with Feature Adapters
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,197
“Although CoOp only uses a small number of parameters, its learnable prompts are equipped in front of the text encoder and require both the forward and backward propagation, as shown by green dotted lines in Figure 1.”
InfluentialDomain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315
Self-citationGrounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,086
“Different from CLIP that fuses vision and language only at the last dot product layer [45], we show that deep cross-modality fusion applied by GLIP, as shown in Figure 2 (Middle), is crucial to learn high-quality language-aware visual representations and to achieve superior transfer learning performance.”
BackgroundResultInfluentialSelf-citationVision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051
Metadata
- Authors
- Zhou, K., Yang, J., Loy, C. C., & Liu, Z.
- Affiliations
- Nanyang Technological University (SG)
- Corresponding author
- Zhou, Kaiyang
- Venue
- International Journal of Computer Vision
- Published
- 2022-07-31
- Volume
- 130
- Issue
- 9
- Pages
- 2337-2348
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W3198377975
- Semantic Scholar
- 96ea07447d2f9adefe03852a878517a2a6d45b96
- Cited by
- 3,304
- References
- 66
- Citations by year
- 2026: 998 · 2025: 1127 · 2024: 789 · 2023: 348 · 2022: 40
- FWCI
- 189.55
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- International Journal of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- prompt learning, vision-language models, CLIP, image classification, zero-shot transfer, few-shot learning, domain generalization, prompt engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.