The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index; none of the 20 sources its claims cite is checked for support yet.
Learning to Prompt for Vision-Language Models
Open accessAbstract
Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming—one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt’s context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.
Overview
CoChat Summary
TL;DR
CoOp automates prompt engineering for CLIP-like vision-language models by learning continuous context vectors while keeping all pre-trained parameters frozen. Across 11 datasets, it turns CLIP into a strong few-shot learner, beating hand-crafted prompts and often outperforming linear probes, while also improving robustness to some domain shifts.
Key findings
- With as few as 1–2 shots, CoOp beats hand-crafted prompts; with 16 shots the average gain is around 15%, with the highest reaching over 45% (EuroSAT >45%, DTD >20%).
- CoOp outperforms the linear probe baseline overall: the linear probe needs more than 4 shots on average to match zero-shot CLIP, while CoOp already shows an impressive average gain at 4 shots.
- On domain generalization from ImageNet, CoOp improves over zero-shot CLIP on several shifted sets, e.g. ResNet-50 source ImageNet 62.95 vs. 58.18, ImageNetV2 55.11 vs. 51.34, ImageNet-A 22.12 vs. 21.65; linear probe is much worse on target datasets.
- Unified context generally performs better, while class-specific context is preferred for some fine-grained and specialized tasks at 16 shots but mostly underperforms unified context with fewer than 8 shots.
Methods
- CoOp models a prompt’s context words as learnable vectors, with two implementations: unified context shared across classes and class-specific context; the class token can be placed at the end or in the middle, and training minimizes cross-entropy while freezing CLIP.
- Experiments use 11 image classification datasets under CLIP’s few-shot protocol with 1, 2, 4, 8, and 16 shots, full test sets, and averages over three runs; default backbone is ResNet-50, context length M=16, SGD with initial learning rate 0.002 and cosine annealing.
- Baselines are zero-shot CLIP with hand-crafted prompts and a linear probe; additional analyses cover context length, vision backbones, prompt ensembling, initialization, and domain generalization on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
Limitations
- The paper states that CoOp’s results are relatively difficult to interpret, like other continuous prompt learning methods, and that CoOp is sensitive to noisy labels, as shown by weak performance on Food101.
- The paper also notes that interpreting learned prompts via nearest vocabulary words could be inaccurate, and that there is no golden rule for selecting context length because one must balance performance and robustness to distribution shift.
- Evident from the text: class-specific context mostly underperforms unified context in low-data scenarios with fewer than 8 shots, and Table 1 shows CoOp is not uniformly better than zero-shot CLIP on every target set, e.g. ResNet-50 with M=16 on ImageNet-R is 54.96 vs. zero-shot 56.00.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 15 Not checked · 67 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe that for pre-trained vision-language models, the text input, known as prompt, plays a key role in downstream datasets.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We later show that CoOp obtains much stronger performance than the linear probe model in domain generalization.”
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe that when using manual initialization (like 'a photo of a'), the nearest words for the converged vectors are mostly the ones used for initialization.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average results are shown in Figure 5(a), which indicate that having more context tokens leads to better performance and that positioning the class token in the middle gains more momentum with longer context length.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given 16 shots for training, the average gap brought by CoOp can be further increased to around 15%.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
46 more claims
Create a free account to see all 84 claims and to verify the 20 references still open.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 17
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“prompt learning methods [2,5,8,16,18,32,42] emerge recently as efficient and effective solutions for adapting pre-trained transformers to the target domain via only training very few parameters (i.e., prompts), and achieve comparable performance with finetuning the whole heavy model.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [42]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ B Results on DOSCO-2k · p. 11
“All models are trained on the training set and the checkpoints with the best validation performance are used for final test in unseen domains.”
§ References · p. 12
“Among the three approaches, CoOp and its follow-up, CoCoOp, contain learnable components while CLIP here denotes the zero-shot model.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Few-Shot Learning · p. 8
“| Zero-Shot CLIP | 61.62 | 54.81 | 38.71 | 28.05 | 64.38 |”
§ 4.2 Domain Generalization · p. 9
“| Prompt engineering | 58.18 | 61.26 | 62.05 | 66.73 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“A common approach for building state-of-the-art visual recognition systems is to train vision models to predict for a fixed set of object categories using discrete labels (He et al., 2016; Dosovitskiy et al., 2021).”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Datasets We select 11 publicly available image classification datasets used in CLIP: ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei et al.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 4×
“, 2016) or ViT (Dosovitskiy et al., 2021)—with a fixed set of weights that are seen as visual concepts and initialized randomly.”
BackgroundMethodInfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“There are some differences that distinguish our approach from the prompt learning methods developed in NLP for language models (e.g., GPT-3 (Brown et al., 2020)).”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 19×
“We briefly introduce vision-language pre-training with a particular focus on CLIP (Radford et al., 2021).”
BackgroundMethodInfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 2×
“, 2017), ii) large-minibatch contrastive representation learning (Chen et al., 2020; He et al., 2020; Hénaff et al., 2020), and iii) web-scale training”
BackgroundUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · cited 4×
“For generic objects (ImageNet & Caltech101), scenes (SUN397) and actions (UCF101), using unified context is clearly better.”
BackgroundInfluential
Research works printed but not linked: 2
- Bossard, L., Guillaumin, M., & Van Gool, L. (2014). Food-101–mining discriminative components with random forests. In ECCV · DOI · Paper page
- Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021b). Gpt understands, too. arXiv preprint arXiv:2103.10385 · arXiv
Cited by
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,796
“[62], recent successes in visionlanguage models are mainly attributed to the developments in i) Transformers [48], ii) contrastive representation learning [4, 17, 20], and iii) web-scale training datasets [24, 40].”
BackgroundMethodInfluentialSelf-citationCLIP-Adapter: Better Vision-Language Models with Feature Adapters
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,197
“Although CoOp only uses a small number of parameters, its learnable prompts are equipped in front of the text encoder and require both the forward and backward propagation, as shown by green dotted lines in Figure 1.”
InfluentialDomain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315
Self-citationGrounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,086
“Different from CLIP that fuses vision and language only at the last dot product layer [45], we show that deep cross-modality fusion applied by GLIP, as shown in Figure 2 (Middle), is crucial to learn high-quality language-aware visual representations and to achieve superior transfer learning performance.”
BackgroundResultInfluentialSelf-citationVision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051
Metadata
- Authors
- Zhou, K., Yang, J., Loy, C. C., & Liu, Z.
- Affiliations
- Nanyang Technological University (SG)
- Corresponding author
- Zhou, Kaiyang
- Venue
- International Journal of Computer Vision
- Published
- 2022-07-31
- Volume
- 130
- Issue
- 9
- Pages
- 2337-2348
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W3198377975
- Semantic Scholar
- 96ea07447d2f9adefe03852a878517a2a6d45b96
- Cited by
- 3,304
- References
- 66
- Citations by year
- 2026: 998 · 2025: 1127 · 2024: 789 · 2023: 348 · 2022: 40
- FWCI
- 189.55
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- International Journal of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- prompt learning, vision-language models, CLIP, image classification, zero-shot transfer, few-shot learning, domain generalization, prompt engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Learning to Prompt for Vision-Language Models. https://app.cochat.ai/papers/10.1007/s11263-022-01653-1
- The paper
- Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022). Learning to Prompt for Vision-Language Models. International Journal of Computer Vision. https://doi.org/10.1007/s11263-022-01653-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.