Prefix Conditioning Unifies Language and Label Supervision
Open accessAbstract
Image-classification datasets have been used to pretrain image recognition models. Recently, web-scale image-caption datasets have emerged as a source of powerful pretraining alternative. Image-caption datasets are more ``open-domain'', containing a wider variety of scene types and vocabulary words than traditional classification datasets, and models trained on these datasets have demonstrated strong performance on few- and zero-shot recognition tasks. When naively unifying image-classification and -caption dataset, we show that such dataset biases negatively affect pre-training by reducing the generalizability of learned representations and thus jeopardizing zero-shot performance since the unification can tailor the model for the classification dataset, making it vulnerable to the distribution shift from the dataset. In this work, we address the problem by disentangling the dataset bias using prefix tokens that inform a language encoder of the type of the input dataset (e.g., image-classification or caption) at training time. This approach allows the language encoder to share the knowledge from two datasets as well as switch the mode of feature extraction, i.e., image-classification dataset or image-caption dataset tailored mode, where we use image-caption mode in the zero-shot evaluation. Our method is generic and can be easily integrated into existing VL pre-training objectives such as CLIP or UniCL. In experiments, we show that this simple technique improves the performance in zero-shot image recognition accuracy and robustness to the image-level distribution shift.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prompts are good indicators for different distributions of input.
“As motivated by [29] which shows that prompts are good indicators for different distributions of input”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 3× · see citing works
“KLite [34] utilizes external knowledge from WordNet [28] and Wikitionary [27].”
MethodInfluentialDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569 · see citing works
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965 · see citing works
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,918 · see citing works
CoCa: Contrastive Captioners are Image-Text Foundation ModelsPreprint
Yu, J.-H., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y.-H. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.01917 · Cited by 1,899 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · see citing works
(2016) · neural information processing systems · Cited by 1,572 · see citing works
Self-citationMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250 · see citing works
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionPreprint
Kumar, A., Raghunathan, A., Jones, R., Ma, T.-Y., & Liang, P. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2202.10054 · Cited by 1,110 · see citing works
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · see citing works
Evaluation of output embeddings for fine-grained image classification
Akata, Z., Reed, S., Walter, D., Lee, H., & Schiele, B. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298911 · Cited by 1,011 · see citing works
Zero-Shot Learning — The Good, the Bad and the Ugly
Xian, Y., Schiele, B., & Akata, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.328 · Cited by 891 · see citing works
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790 · see citing works
Latent Embeddings for Zero-Shot Classification
Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., & Schiele, B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.15 · Cited by 723 · see citing works
Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs
Wang, X., Ye, Y., & Gupta, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00717 · Cited by 659 · see citing works
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
FILIP: Fine-grained Interactive Language-Image Pre-TrainingPreprint
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 388 · see citing works
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
Learning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Xing, E. P., & Lipton, Z. C. (2019) · arXiv · doi:10.48550/arxiv.1905.13549 · Cited by 337 · see citing works
Vision-Language Pre-Training with Triple Contrastive Learning
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., & Huang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01522 · Cited by 309 · see citing works
(2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 291 · see citing works
SLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 285 · see citing works
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training ParadigmPreprint
Li, Y., Liang, F., Zhao, L.-C., Cui, Y., Ouyang, W., Shao, J., Yu, F., & Yan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05208 · Cited by 264 · see citing works
Zero Shot Recognition with Unreliable AttributesPreprint
Jayaraman, D., & Grauman, K. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.4327 · Cited by 240 · see citing works
Unified Contrastive Learning in Image-Text-Label Space
Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., & Gao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01857 · Cited by 202 · see citing works
Self-citationCOSTA: Co-Occurrence Statistics for Zero-Shot Classification
Mensink, T., Gavves, E., & Snoek, C. G. M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.313 · Cited by 194 · see citing works
Learning De-biased Representations with Biased RepresentationsPreprint
Bahng, H., Chun, S., Yun, S., Choo, J., & Oh, S. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02806 · Cited by 146 · see citing works
Adaptive Methods for Real-World Domain Generalization
Dubey, A., Ramanathan, V., Pentland, A., & Mahajan, D. K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01411 · Cited by 86 · see citing works
Learning Debiased Representation via Disentangled Feature AugmentationPreprint
Lee, J., Kim, E., Lee, J. Y., Lee, J., & Choo, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01372 · Cited by 66 · see citing works
K-LITE: Learning Transferable Visual Models with External Knowledge
Shen, S., Li, C., Hu, X., Xie, Y., Yang, J., Zhang, P., Gan, Z., Wang, L., Yuan, L., Liu, C., et al. (2022) · neural information processing systems · doi:10.52202/068431-1132 · Cited by 57 · see citing works
Self-citation
Flagged first, then by citations
4 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Stephanie CY Chan, Adam Santoro, Andrew K Lampinen, Jane X Wang, Aaditya Singh, Pierre H Richemond, Jay Mc- Clelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05055, 2022. 1 · arXiv
- Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, and Osamu Yoshie. Zerovl: A strong baseline for align- ing vision-language representations with limited resources. arXiv preprint arXiv:2112.09331, 2021. 4, 5 · arXiv
- Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. arXiv preprint arXiv:2203.12119, 2022. 3 · arXiv
- Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 5
Cited by
Sorted by citations
Multimodal Prompting with Missing Modalities for Visual Recognition
Lee, Y.-L., Tsai, Y.-H., Chiu, W.-C., & Lee, C.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01435 · Cited by 156 · see citing works
Self-citationLearning Customized Visual Models with Retrieval-Augmented Knowledge
Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., & Li, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01454 · Cited by 41 · see citing works
Self-citationYu, B. X. B., Chang, J., Wang, H., Liu, L., Wang, S., Wang, Z., Lin, J., Xie, L., Li, H., Lin, Z., et al. (2024) · ACM Computing Surveys · doi:10.1145/3657632 · Cited by 43 · see citing works
Guo, J., Wang, M., Yin, H., Song, B., Chi, Y., Yu, F. R., & Yuen, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3496491 · Cited by 38 · see citing works
Defense-Prefix for Preventing Typographic Attacks on CLIP
Azuma, H., & Matsui, Y. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00392 · Cited by 15 · see citing works
Medical Language Mixture of Experts for Improving Medical Image Segmentation
Liu, Y., Pei, J., Zhu, H., Yang, G., Jiang, Z., & Lao, Q. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822440 · Cited by 8 · see citing works
Boosting Speech Recognition Robustness to Modality-Distortion with Contrast-Augmented Prompts
Fu, D.-J., Cheng, X.-Z., Yang, X.-D., Wang, H., Zhao, Z., & Jin, T. (2024) · ACM Multimedia · doi:10.1145/3664647.3681347 · Cited by 22 · see citing works
LM-CLIP: Adapting Positive Asymmetric Loss for Long-Tailed Multi-Label Classification
Timmermann, C., Jung, S., Kim, M., & Lee, W. (2025) · IEEE Access · doi:10.1109/access.2025.3561581 · Cited by 4 · see citing works
Chen, C., Chen, W., Lin, Y., Li, B., & Huang, J. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744809 · Cited by 2 · see citing works
Modality Adaptive Network for Arbitrary Modality Salient Object Detection
Yang, Y., Huang, N., Zhang, Q., Han, J., & Huang, J. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3651119 · Cited by 2 · see citing works
Offline prompt reinforcement learning method based on feature extraction
Yao, T., Chen, X., Yao, Y., Huang, W., & Chen, Z. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2490 · Cited by 2 · see citing works
Multi-rater Prompting for Ambiguous Medical Image Segmentation
Wang, J., Cheng, Y., Chen, J., Xu, H., Chen, D., & Wu, J. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822080 · Cited by 2 · see citing works
Entity-Aware Cross-Modal Pretraining for Knowledge-Based Visual Question Answering
Adjali, O., Ferret, O., Ghannay, S., & Borgne, H. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_38 · Cited by 1 · see citing works
Noise-Augmented Missing Modality Aware Prompt Based Learning for Robust Visual Recognition
Wang, Y., Jang, J., & Kim, C. (2023) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip59821.2023.10402690
Class-Customized Domain Adaptation: Unlock Each Customer-Specific Class With Single Annotation
Chen, K., Chang, H., Xu, M., Du, R., Wu, M., Ma, Z., & Zhang, C. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3597036
Fu, D., Cheng, X., Chen, J., Jin, T., & Zhang, Z. M. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3682022
Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data
Phung, T. H., Nguyen, D. M., Huynh, T. T., Nguyen, Q. V. H., Hoang, T. N., & Nguyen, P. L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00375
- Authors
- Saito, K., Sohn, K., Zhang, X., Li, C.-L., Lee, C.-Y., Saenko, K., & Pfister, T.
- Affiliations
- Boston University (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 2861-2870
- ISSN
- 1063-6919 (ISSN-L)
- arXiv
- 2206.01125
- OpenAlex
- W4386075949
- Cited by
- 17
- References
- 48
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 5 · 2023: 4
- FWCI
- 1.82
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- data bias, distribution shift, CLIP, multimodal pre-training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
