Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index; none of the 20 sources its claims cite is checked for support yet.

Learning to Prompt for Vision-Language Models

Open access
Zhou, Kaiyang, Yang, Jingkang, Loy, Chen Change, Liu, Ziwei·2022·International Journal of Computer Vision

Abstract

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming—one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt’s context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

Overview

CoChat Summary

TL;DR

CoOp automates prompt engineering for CLIP-like vision-language models by learning continuous context vectors while keeping all pre-trained parameters frozen. Across 11 datasets, it turns CLIP into a strong few-shot learner, beating hand-crafted prompts and often outperforming linear probes, while also improving robustness to some domain shifts.

Key findings

  • With as few as 1–2 shots, CoOp beats hand-crafted prompts; with 16 shots the average gain is around 15%, with the highest reaching over 45% (EuroSAT >45%, DTD >20%).
  • CoOp outperforms the linear probe baseline overall: the linear probe needs more than 4 shots on average to match zero-shot CLIP, while CoOp already shows an impressive average gain at 4 shots.
  • On domain generalization from ImageNet, CoOp improves over zero-shot CLIP on several shifted sets, e.g. ResNet-50 source ImageNet 62.95 vs. 58.18, ImageNetV2 55.11 vs. 51.34, ImageNet-A 22.12 vs. 21.65; linear probe is much worse on target datasets.
  • Unified context generally performs better, while class-specific context is preferred for some fine-grained and specialized tasks at 16 shots but mostly underperforms unified context with fewer than 8 shots.

Methods

  • CoOp models a prompt’s context words as learnable vectors, with two implementations: unified context shared across classes and class-specific context; the class token can be placed at the end or in the middle, and training minimizes cross-entropy while freezing CLIP.
  • Experiments use 11 image classification datasets under CLIP’s few-shot protocol with 1, 2, 4, 8, and 16 shots, full test sets, and averages over three runs; default backbone is ResNet-50, context length M=16, SGD with initial learning rate 0.002 and cosine annealing.
  • Baselines are zero-shot CLIP with hand-crafted prompts and a linear probe; additional analyses cover context length, vision backbones, prompt ensembling, initialization, and domain generalization on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.

Limitations

  • The paper states that CoOp’s results are relatively difficult to interpret, like other continuous prompt learning methods, and that CoOp is sensitive to noisy labels, as shown by weak performance on Food101.
  • The paper also notes that interpreting learned prompts via nearest vocabulary words could be inaccurate, and that there is no golden rule for selecting context length because one must balance performance and robustness to distribution shift.
  • Evident from the text: class-specific context mostly underperforms unified context in low-data scenarios with fewer than 8 shots, and Table 1 shows CoOp is not uniformly better than zero-shot CLIP on every target set, e.g. ResNet-50 with M=16 on ImageNet-R is 54.96 vs. zero-shot 56.00.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11263-022-01653-1

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 15 Not checked · 67 Own results and interpretations

46 more claims

Create a free account to see all 84 claims and to verify the 20 references still open.

See all claims — free account
  • It may state two separate claims as one. 19
  • Probably not a substantial claim. 17
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 66 references

Research works printed but not linked: 2

  1. Bossard, L., Guillaumin, M., & Van Gool, L. (2014). Food-101–mining discriminative components with random forests. In ECCV · DOI · Paper page
  2. Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021b). Gpt understands, too. arXiv preprint arXiv:2103.10385 · arXiv

Cited by

  • Conditional Prompt Learning for Vision-Language Models

    Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,796

    “[62], recent successes in visionlanguage models are mainly attributed to the developments in i) Transformers [48], ii) contrastive representation learning [4, 17, 20], and iii) web-scale training datasets [24, 40].”
    BackgroundMethodInfluentialSelf-citation
  • CLIP-Adapter: Better Vision-Language Models with Feature Adapters

    Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,197

    “Although CoOp only uses a small number of parameters, its learnable prompts are equipped in front of the text encoder and require both the forward and backward propagation, as shown by green dotted lines in Figure 1.”
    Influential
  • Domain Generalization: A Survey

    Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315

    Self-citation
  • Grounded Language-Image Pre-training

    Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,086

    “Different from CLIP that fuses vision and language only at the last dot product layer [45], we show that deep cross-modality fusion applied by GLIP, as shown in Figure 2 (Middle), is crucial to learn high-quality language-aware visual representations and to achieve superior transfer learning performance.”
    BackgroundResultInfluentialSelf-citation
  • Vision-Language Models for Vision Tasks: A Survey

    Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051

See all 3,304

Metadata

Authors
Zhou, K., Yang, J., Loy, C. C., & Liu, Z.
Affiliations
Nanyang Technological University (SG)
Corresponding author
Zhou, Kaiyang
Venue
International Journal of Computer Vision
Published
2022-07-31
Volume
130
Issue
9
Pages
2337-2348
ISSN
  • 0920-5691 (ISSN-L)
  • 1573-1405
OpenAlex
W3198377975
Cited by
3,304
References
66
Citations by year
2026: 998 · 2025: 1127 · 2024: 789 · 2023: 348 · 2022: 40
FWCI
189.55
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computer Vision (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DR-NTU (Nanyang Technological University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
prompt learning, vision-language models, CLIP, image classification, zero-shot transfer, few-shot learning, domain generalization, prompt engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning to Prompt for Vision-Language Models. https://app.cochat.ai/papers/10.1007/s11263-022-01653-1
The paper
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022). Learning to Prompt for Vision-Language Models. International Journal of Computer Vision. https://doi.org/10.1007/s11263-022-01653-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.