The DOI matches its Crossref record; 194 of the 202 references it cites were found in a registry or an index.
Vision-Language Models for Vision Tasks: A Survey
Open accessAbstract
Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With the rise of multi-modal forgeries, recent approaches leverage deep learning models such as transformers [8] and visionlanguage models [9] to improve detection capabilities.”
From Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges (2024) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As LLMs expand in the kinds of stimuli they can interpret -e.g.visual (OpenAI, 2023b;Zhang, Huang, et al., 2024;Gemini Team et al., 2024) -and the ways in which they can interact with the world -e.g.embodiment and tool use (Mialon et al., 2023) -, the space of transferable paradigms increases.”
From Machine Psychology (2023) · cites this paper as Zhang, Huang, et al., 2024
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Regarding improvements in symbolic or numeric reasoning, another technique is to prompt LLMs to use code for solving tasks (Zhang, Ge, et al., 2024).”
From Machine Psychology (2023) · cites this paper as Zhang, Ge, et al., 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 1×
“Given a batch of B images, contrastive-learning objectives ( e.g. , InfoNCE [68] and its variants [12], [13]) are usually formulated as follows: where z Ii is the query embedding, { z Ij } B +1 j =1 ,j ̸ = i are key embeddings, where z I + stands for z Ii ’s positive key and the rest are z Ii ’s…”
BackgroundVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 1×
“• VG [76] provides a multi-perspective understanding of images, e.g. , object-level information, scene graphs and visual question answer pairs.”
BackgroundSegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 3×
“Different from prompt engineering [165] that manually designs text prompts for each task, text prompt tuning explores more effective and efficient learnable text prompts with several labelled down-stream samples for each class.”
BackgroundMethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 3×
“Different from prompt engineering [165] that manually designs text prompts for each task, text prompt tuning explores more effective and efficient learnable text prompts with several labelled down-stream samples for each class.”
BackgroundMethodReading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442
Scene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283
Remote Sensing Image Scene Classification: Benchmark and State of the Art
Cheng, G., Han, J., & Lu, X. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2675998 · Cited by 2,926
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · Cited by 2,760
Peterson, L. E. (2009) · Scholarpedia · doi:10.4249/scholarpedia.1883 · Cited by 2,596
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.383 · Cited by 2,513
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270
Food-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Gool, L. V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,105
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 2,069
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
Helber, P., Bischke, B., Dengel, A., & Borth, D. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2918242 · Cited by 1,951
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,715
Parkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,702
Bag of Tricks for Image Classification with Convolutional Neural Networks
He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00065 · Cited by 1,680
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,589
Open-vocabulary Object Detection via Vision and Language Knowledge DistillationPreprint
Gu, X.-Y., Lin, T.-Y., Kuo, W.-C., & Cui, Y. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2104.13921 · Cited by 1,464
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438
CoCa: Contrastive Captioners are Image-Text Foundation ModelsPreprint
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01917 · Cited by 1,413
BEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,336
LVIS: A Dataset for Large Vocabulary Instance Segmentation
Gupta, A., Dollar, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00550 · Cited by 1,274
Fine-Grained Visual Classification of AircraftPreprint
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., & Vedaldi, A. (2013) · HAL · doi:10.48550/arxiv.1306.5151 · Cited by 1,250
The German Traffic Sign Recognition Benchmark: A multi-class classification competition
Stallkamp, J., Schlipsing, M., Salmen, J., & Igel, C. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033395 · Cited by 1,242
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01832 · Cited by 1,125
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Cheng, G., Xie, X., Han, J., Guo, L., & Xia, G.-S. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3005403 · Cited by 1,059
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
Scene Text Recognition using Higher Order Language Priors
Mishra, A., Alahari, K., & Jawahar, C. V. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.127 · Cited by 851
Deep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 825
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 802
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., & Lu, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01755 · Cited by 718
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 701
RegionCLIP: Region-based Language-Image Pretraining
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01629 · Cited by 681
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 678
Objects365: A Large-Scale, High-Quality Dataset for Object Detection
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00852 · Cited by 673 · cited 3×
“With the object locating ability learned from auxiliary datasets [85], [86], pre-trained VLMs achieve zero-shot prediction for object detection tasks by comparing the embeddings of the given object proposals and texts.”
BackgroundMethodReproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673
Image Segmentation Using Text and Image Prompts
Lüddecke, T., & Ecker, A. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00695 · Cited by 657
A Corpus for Reasoning about Natural Language Grounded in Photographs
Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 634
Making Convolutional Networks Shift-Invariant AgainPreprint
Zhang, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11486 · Cited by 611
FILIP: Fine-grained Interactive Language-Image Pre-TrainingPreprint
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 595
Detecting Twenty-Thousand Classes Using Image-Level Supervision
Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., & Misra, I. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_21 · Cited by 564
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., & Marculescu, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00682 · Cited by 553
PointCLIP: Point Cloud Understanding by CLIP
Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, Y., Gao, P., & Li, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00836 · Cited by 538
GroupViT: Semantic Segmentation Emerges from Text Supervision
Xu, J., Mello, S. D., Liu, S., Byeon, W., Breuel, T. M., Kautz, J., & Wang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01760 · Cited by 518
Extract Free Dense Labels from CLIP
Zhou, C., Loy, C. C., & Dai, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_40 · Cited by 513
Rotation Equivariant CNNs for Digital Pathology
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., & Welling, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00934-2_24 · Cited by 476
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language ModelingPreprint
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03930 · Cited by 472
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
Ghiasi, G., Gu, X., Cui, Y., & Lin, T.-Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_31 · Cited by 446
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428
VLMo: Unified Vision-Language Pre-Training with Mixture-Of-Modality-Experts
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., & Wei, F. (2022) · neural information processing systems · doi:10.52202/068431-2384 · Cited by 420
Simple Open-Vocabulary Object Detection
Minderer, M., Gritsenko, A. A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_42 · Cited by 406
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402
Multiclass and Binary SVM Classification: Implications for Training and Classification Users
Mathur, A., & Foody, G. M. (2008) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2008.915597 · Cited by 394
Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., & Li, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01369 · Cited by 393
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training ParadigmPreprint
Li, Y., Liang, F., Zhao, L.-C., Cui, Y., Ouyang, W., Shao, J., Yu, F., & Yan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05208 · Cited by 391
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389
Prompt-aligned Gradient for Prompt Tuning
Zhu, B., Niu, Y., Han, Y., Wu, Y., & Zhang, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01435 · Cited by 387
SLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 385
Side Adapter Network for Open-Vocabulary Semantic Segmentation
Xu, M., Zhang, Z., Wei, F., Hu, H., & Bai, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00288 · Cited by 368
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358
The hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352
Decoupling Zero-Shot Semantic Segmentation
Ding, J., Xue, N., Xia, G.-S., & Dai, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01129 · Cited by 339
A Short Note on the Kinetics-700-2020 Human Action Dataset
Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.10864 · Cited by 334
Deep learning in robotics: A review of recent research
Pierson, H. A., & Gashler, M. S. (2017) · Advanced Robotics · doi:10.1080/01691864.2017.1365009 · Cited by 331
VLP: A Survey on Vision-language Pre-training
Chen, F., Zhang, D., Han, M., Chen, X.-Y., Shi, J. C., Xu, S., & Xu, B. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1369-5 · Cited by 330
Language-driven Semantic Segmentation
Li, B., Weinberger, K. Q., Belongie, S., Koltun, V., & Ranftl, R. (2022) · arXiv · doi:10.48550/arxiv.2201.03546 · Cited by 316
Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds
Berg, T., Liu, J., Lee, S. W., Alexander, M. L., Jacobs, D. W., & Belhumeur, P. N. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.259 · Cited by 313
A Survey of Vision-Language Pre-Trained Models
Du, Y., Liu, Z., Li, J., & Zhao, W. X. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/762 · Cited by 313
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization
Yao, H., Zhang, R., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00653 · Cited by 310
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model
Xu, M., Zhang, Z., Wei, F., Lin, Y., Cao, Y., Hu, H., & Bai, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19818-2_42 · Cited by 296
What does a platypus look like? Generating customized prompts for zero-shot image classification
Pratt, S., Covert, I., Liu, R., & Farhadi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01438 · Cited by 289
Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., & He, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01469 · Cited by 287
Lu, Y., Liu, J., Zhang, Y., Liu, Y., & Tian, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00514 · Cited by 285
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 272
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation
Zhou, Z., Lei, Y., Zhang, B., Liu, L., & Liu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01075 · Cited by 270
Open-Vocabulary DETR with Conditional Matching
Zang, Y., Li, W., Zhou, K., Huang, C., & Loy, C. C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20077-9_7 · Cited by 254
Semantic Projection Network for Zero- and Few-Label Semantic Segmentation
Xian, Y., Choudhury, S., He, Y., Schiele, B., & Akata, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00845 · Cited by 248
Visual Classification via Description from Large Language ModelsPreprint
Menon, S., & Vondrick, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07183 · Cited by 247
Chinese CLIP: Contrastive Vision-Language Pretraining in ChinesePreprint
An, Y., Pan, J., Lin, J., Men, R., Zhang, Y., Zhou, J., & Zhou, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.01335 · Cited by 237
PaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235
Unified Contrastive Learning in Image-Text-Label Space
Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., & Gao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01857 · Cited by 216
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., & Najork, M. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463257 · Cited by 214
CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation
Xie, J., Hou, X., Ye, K., & Shen, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00444 · Cited by 213
PromptDet: Towards Open-Vocabulary Detection Using Uncurated Images
Feng, C., Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X. K., Xie, W., & Ma, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20077-9_41 · Cited by 191
CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training
Huang, T., Dong, B., Yang, Y., Huang, X., Lau, R. W. H., Ouyang, W., & Zuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02025 · Cited by 191
Connecting Vision and Language with Localized Narratives
Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., & Ferrari, V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_38 · Cited by 188
Unsupervised Prompt Learning for Vision-Language ModelsPreprint
Huang, T. J., Chu, J., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03649 · Cited by 165
Aligning Bag of Regions for Open-Vocabulary Object Detection
Wu, S., Zhang, W., Jin, S., Liu, W., & Loy, C. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01464 · Cited by 155
Self-citationCALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention
Guo, Z., Zhang, R., Qiu, L., Ma, X., Miao, X., He, X., & Cui, B. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25152 · Cited by 143
Task Residual for Tuning Vision-Language Models
Yu, T., Lu, Z., Jin, X., Chen, Z., & Wang, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01049 · Cited by 143
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language ModelsPreprint
Shu, M., Nie, W., Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., & Xiao, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.07511 · Cited by 140
Exploring Visual Prompts for Adapting Large-Scale Models
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022) · arXiv · doi:10.48550/arxiv.2203.17274 · Cited by 129
Image-text Retrieval: A Survey on Recent Research and Development
Cao, M., Li, S., Li, J., Nie, L., & Zhang, M. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/759 · Cited by 125
SuS-X: Training-Free Name-Only Transfer of Vision-Language Models
Udandarao, V., Gupta, A., & Albanie, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00257 · Cited by 120
RedCaps: web-curated image-text data created by the people, for the peoplePreprint
Desai, K., Kaul, G., Aysola, Z., & Johnson, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11431 · Cited by 119
Improving CLIP Training with Language Rewrites
Fan, L., Krishnan, D., Isola, P., Katabi, D., & Tian, Y. (2023) · neural information processing systems · doi:10.52202/075280-1544 · Cited by 112
Peng, F., Yang, X., Xiao, L., Wang, Y., & Xu, C. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3311646 · Cited by 110
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
Chen, Z., Liu, G., Zhang, B., Yang, Q., & Wu, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.552 · Cited by 109
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y. J., et al. (2022) · neural information processing systems · doi:10.52202/068431-0675 · Cited by 109
Unified Vision and Language Prompt LearningPreprint
Zang, Y., Li, W., Zhou, K., Huang, C., & Loy, C. C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07225 · Cited by 107
FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Oin, J., Wu, J., Yan, P., Li, M., Yuxi, R., Xiao, X., Wang, Y., Wang, R., Wen, S., Pan, X., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01863 · Cited by 106
Exploiting Unlabeled Data with Vision and Language Models for Object Detection
Zhao, S., Zhang, Z.-X., Schulter, S., Zhao, L., Kumar, B. G. V., Stathopoulos, A., Chandraker, M. K., & Metaxas, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20077-9_10 · Cited by 104
Collecting a Large-scale Dataset of Fine-grained Cars
Krause, J., Deng, J., Stark, M. M., & Fei-Fei, L. (2013) · MPG.PuRe (Max Planck Society) · Cited by 100
Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
Gao, M., Chen, X., Niebles, J. C., Li, J., Xu, R., Liu, W., & Xiong, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_16 · Cited by 95
Coarse-to-fine vision-language pre-training with fusion in the backbone
Dou, Z.-Y., Kamath, A., Gan, Z., Zhang, P., Wang, J., Li, L., Liu, Z., Liu, C., Lecun, Y., Peng, N., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2387 · Cited by 93
Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling
Huynh, D., Kuen, J., Lin, Z., Gu, J., & Elhamifar, E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00689 · Cited by 92
Learning Object-Language Alignments for Open-Vocabulary Object DetectionPreprint
Lin, C., Sun, P., Jiang, Y., Luo, P., Qu, L., Haffari, G., Yuan, Z., & Cai, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.14843 · Cited by 90
Bridging the Gap between Object and Image-level Representations for Open-Vocabulary DetectionPreprint
Rasheed, H., Maaz, M., Khattak, M. U., Khan, S., & Khan, F. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.03482 · Cited by 89
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection
Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., & Xu, H. (2022) · arXiv · doi:10.48550/arxiv.2209.09407 · Cited by 86
Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection
Wang, L., Liu, Y., Du, P., Ding, Z., Liao, Y., Qi, Q., Chen, B., & Liu, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01076 · Cited by 85
Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
Kim, D., Angelova, A., & Kuo, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01072 · Cited by 85
Multitask Vision-Language Prompt Tuning
Shen, S., Yang, S., Zhang, T., Zhai, B., Gonzalez, J. E., Keutzer, K., & Darrell, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00556 · Cited by 85
DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations
Sun, X., Hu, P., & Saenko, K. (2022) · neural information processing systems · doi:10.52202/068431-2216 · Cited by 85
Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models
Ma, C., Liu, Y., Deng, J., Xie, L., Dong, W., & Xu, C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3245584 · Cited by 84
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language ModelsPreprint
Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., & Angelova, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15639 · Cited by 84
Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation
Mal, Z., Luo, G., Gao, J. G., Li, L., Chen, Y., Wang, S., Zhang, C., & Hu, W. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01368 · Cited by 74
Texts as Images in Prompt Tuning for Multi-Label Image Recognition
Guo, Z., Dong, B., Ji, Z., Bai, J., Guo, Y., & Zuo, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00275 · Cited by 73
K-LITE: Learning Transferable Visual Models with External Knowledge
Shen, S., Li, C., Hu, X., Xie, Y., Yang, J., Zhang, P., Gan, Z., Wang, L., Yuan, L., Liu, C., et al. (2022) · neural information processing systems · doi:10.52202/068431-1132 · Cited by 68
SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic SegmentationPreprint
Luo, H., Bao, J., Wu, Y., He, X., & Li, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.14813 · Cited by 68
CLIPPO: Image-and-Language Understanding from Pixels Only
Tschannen, M., Mustafa, B., & Houlsby, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01059 · Cited by 65
PyramidCLIP: Hierarchical Feature Alignment for Vision-Language Model Pretraining
Gao, Y., Liu, J., Xu, Z., Zhang, J., Li, K., Ji, R., & Shen, C. (2022) · neural information processing systems · doi:10.52202/068431-2606 · Cited by 60
Self-citationReCo: Retrieve and Co-Segment for Zero-Shot Transfer
Shin, G., Xie, W., & Albanie, S. (2022) · neural information processing systems · doi:10.52202/068431-2446 · Cited by 55
Vision-Language Intelligence: Tasks, Representation Learning, and Large ModelsPreprint
Li, F., Zhang, H., Zhang, Y., Liu, S., Guo, J., Ni, L. M.-S., Zhang, P., & Zhang, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.01922 · Cited by 53
RA-CLIP: Retrieval Augmented Contrastive Language-Image Pre-Training
Xie, C.-W., Sun, S., Xiong, X., Zheng, Y., Zhao, D., & Zhou, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01846 · Cited by 51
ALIP: Adaptive Language-Image Pre-training with Synthetic Caption
Yang, K., Deng, J., An, X., Li, J., Feng, Z., Guo, J., Yang, J., & Liu, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00273 · Cited by 50
SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained ModelsPreprint
Pantazis, O., Brostow, G., Jones, K. E., & Aodha, O. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03794 · Cited by 42
VT-CLIP: Enhancing Vision-Language Models with Visual-guided TextsPreprint
Qiu, L., Zhang, R., Guo, Z., Zeng, Z., Guo, Z., Li, Y., & Zhang, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.02399 · Cited by 39
UniCLIP: Unified Framework for Contrastive Language-Image Pre-Training
Lee, J., Kim, J., Shon, H., Kim, B., Kim, S. H., Lee, H., & Kim, J. (2022) · neural information processing systems · doi:10.52202/068431-0074 · Cited by 35
CPL: Counterfactual Prompt Learning for Vision and Language Models
He, X., Yang, D., Feng, W., Fu, T.-J., Akula, A., Jampani, V., Narayana, P., Basu, S., Wang, W. Y., & Wang, X. E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.224 · Cited by 32
NLIP: Noise-Robust Language-Image Pre-training
Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C., & Liang, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25172 · Cited by 30
Non-Contrastive Learning Meets Language-Image Pre-Training
Zhou, J., Li, D., Gan, Z., Wang, L., & Wei, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01061 · Cited by 30
Fine-Grained Visual–Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
Long, Y., Han, J., Huang, R., Xu, H., Zhu, Y., Xu, C., & Liang, X. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3293484 · Cited by 29
Contrastive Vision-Language Pre-training with Limited Resources
Cui, Q., Zhou, B., Guo, Y. T., Yin, W., Wu, H., Yoshie, O., & Chen, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_14 · Cited by 29
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H. S., Zhang, X.-P. S., & Tang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01282 · Cited by 28
Learning to Detect and Segment for Open Vocabulary Object Detection
Wang, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00681 · Cited by 26
Multi-task prompt tuning with soft context sharing for vision–language models
Ding, K., Wang, Y., Liu, P., Yu, Q., Zhang, H., Xiang, S., & Pan, C. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128290 · Cited by 24
Retrieval-Enhanced Visual Prompt Learning for Few-Shot Classification
Rong, J., Chen, H., Ou, L., Chen, T., Yu, X., & Liu, Y. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3651774 · Cited by 24
Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport DistillationPreprint
Wu, B., Cheng, R., Zhang, P., Gao, T., Vajda, P., & Gonzalez, J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.09445 · Cited by 22
HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware AttentionPreprint
Geng, S., Yuan, J., Tian, Y., Chen, Y., & Zhang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.02995 · Cited by 22
Unifying Vision-Language Representation Space with Single-Tower Transformer
Jang, J., Kong, C., Jeon, D., Kim, S., & Kwak, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25178 · Cited by 18
Large-scale Bilingual Language-Image Contrastive LearningPreprint
Ko, B., & Gu, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.14463 · Cited by 15
RareAct: A video dataset of unusual interactionsPreprint
Miech, A., Alayrac, J.-B., Ivan Laptev, A., Šivic, J., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01018 · Cited by 14
ZSD-YOLO: Zero-Shot YOLO Detection using Vision-Language KnowledgeDistillation.Preprint
Xie, J., & Zheng, S. (2021) · arXiv (Cornell University) · Cited by 13
Semantic Segmentation In-the-Wild Without Seeing Any Segmentation ExamplesPreprint
Zabari, N., & Hoshen, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.03185 · Cited by 11
ProTeCt: Prompt Tuning for Taxonomic Open Set Classification
Wu, T.-Y., Ho, C.-H., & Vasconcelos, N. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01564 · Cited by 10
Improving Zero-Shot Models with Label Distribution PriorsPreprint
Kahana, J., Cohen, N., & Hoshen, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.00784 · Cited by 9
GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training
Deng, X., Shi, H., Huang, R., Li, C., Xu, H., Han, J., Kwok, J. T.-Y., Zhao, S., Zhang, W., & Liang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02027 · Cited by 2
Research works printed but not linked: 5
- 10.5555/3524938.3525087 · DOI
- Radford, 2018, Improving language understanding by generative pre-training, OpenAI Blog
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1
- 10.1109/cvpr52688.2022.01553 · DOI · Paper page
- Ordonez, Im2Text: Describing images using 1 million captioned photographs, 1143
Cited by
Sorted by citations
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing
Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., & Zhou, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390838 · Cited by 507
The Segment Anything Model (SAM) for remote sensing applications: From zero to one shot
Osco, L. P., Wu, Q., de Lemos, E. L., Gonçalves, W. N., Ramos, A. P. M., Li, J., & Marcato, J. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103540 · Cited by 405
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304
Multimodal Alignment and Fusion: A Survey
Li, S., & Tang, H. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02667-1 · Cited by 267
Foundation Models for Remote Sensing and Earth Observation: A survey
Xiao, A., Xuan, W., Wang, J., Huang, J., Tao, D., Lu, S., & Yokoya, N. (2025) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2025.3576766 · Cited by 186
Self-citation
Metadata
- Authors
- Zhang, J., Huang, J., Jin, S., & Lu, S.
- Affiliations
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-08
- Volume
- 46
- Issue
- 8
- Pages
- 5625-5644
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4392172801
- Cited by
- 1,051
- References
- 202
- Citations by year
- 2026: 432 · 2025: 486 · 2024: 124
- FWCI
- 132.68
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- vision-language models, visual recognition, zero-shot learning, transfer learning, knowledge distillation, image-text dataset, pre-training objectives, network architecture, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
