The DOI matches its DataCite record; 185 of the 193 references it cites were found in a registry or an index, and 1 are not checked yet.
Learning Transferable Visual Models From Natural Language Supervision
Open accessAbstract
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“During training,the model learns the joint distribution between the two modalities and optimizes the model through methods such as adversarial training or maximizing likelihood, enabling it to generate the target modality”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [188]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large Image Models (LIMs) such as CLIP [86] and BLIP-2 [87] extend the agent's capabilities into the visual domain. Trained on image-text pairs, LIMs enable perception-based tasks including image classification, object detection, and vision-language grounding.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [86]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Foundation Models (FMs) [23, 35, 37] - generic models trained on web-scale datasets - have demonstrated remarkable zero-shot performance and can be quickly adapted to various downstream tasks with few additional tuning cost.”
From Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (2024) · cites this paper as [37]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By encoding text descriptions of variations of the dataset bias (e.g.'a blue wall'), using CLIP's text encoder (CLIP T ), we vary visual representations without actually having seen images containing these variations (e.g. an image of a blue wall).”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, their capacity for semantic comprehension is comparatively limited when contrasted with vision-based perceptual models like CLIP (Radford et al., 2021).”
From WorldVLA: Towards Autoregressive Action World Model (2025) · cites this paper as Radford et al., 2021
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In natural language [1] and computer vision [39], general-purpose foundation models that are pre-trained on diverse multi-task data tend to outperform narrowly tailored and specialized solutions.”
From $π_0$: A Vision-Language-Action Flow Model for General Robot Control (2024) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing evaluation metrics such as CLIP-Image score [37] and DINO score [6], while commonly used, often fail to reflect subtle differences due to their reliance on high-dimensional semantic features for cosine similarity computation.”
From DiffSim: Taming Diffusion Models for Evaluating Visual Similarity (2024) · cites this paper as [37]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CLIP cosine similarity is often used to measure text prompt and frame consistency, where a reference video dataset is not needed.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Radford et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Trained on image-text pairs, LIMs enable perception-based tasks including image classification, object detection, and vision-language grounding.”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [88]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Vision-Language Models (VLMs) e.g., CLIP [15], allows us to augment at feature-level based on prior knowledge of the environment without any changes to training data.”
From Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation (2023) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“We find these methods perform poorly on language and image goals, particularly when relying on CLIP [25] features.”
From GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“A notable advancement in this domain is the recent work on CLIP [9, 32, 35, 43], which elevates open-vocabulary recognition performance by collecting millions of image-text pairs and employing contrastive learning techniques.”
From Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations (2024) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“The recently proposed Contrastive Language-Image Pretraining (CLIP) [1] learns the correspondence between text and image by projecting them into a shared latent space.”
From Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2023) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“To evaluate the visual similarity of I R and I G , we use the similarity of their CLIP (Radford et al., 2021) embedding, denoted as CLIP ( I R , I G ) .”
From Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025) · cites this paper as Radford et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“This suggests the inability of CLIP [25] features in capturing crucial instance-specific and spatial features in language and image goals.”
From GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Existing solutions in multimodal Transformer adopt a fixed fusion strategy [18,21,24,26,33].”
From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [33]
The cited source says what the claim attributes to it.
In the citing paper
“existing foundation models for vision and language such as CLIP [8], SigLIP [9], and Llama 2 [10] are capable of these types of generalization and more, stemming from the priors captured by their Internet-scale pretraining datasets.”
From OpenVLA: An Open-Source Vision-Language-Action Model (2024) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“which shows that our model achieves the highest DINO [5] and CLIP-I [42] scores, with CLIP-T [42] scores that are comparable to other methods.”
From DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models (2025) · cites this paper as [42]
The cited source says what the claim attributes to it.
In the citing paper
“In early 2021, OpenAI released an architecture termed Contrastive Language Image Pretraining (CLIP), which, when trained on millions of image-text pairs, matched the performance of competitive, fully supervised models without fine-tuning 106”
From Multimodal biomedical AI (2022) · cites this paper as [106]
The cited source says what the claim attributes to it.
In the citing paper
“Even when PwD data is available, data from marginalized groups is often automatically filtered out by techniques for removing 'low' quality data [57, 104], which rely on biased models such as CLIP [92].”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [92]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HOLODECK uses a Large Language Model (GPT-4 [37]) to design the floor plan, assign suitable materials, install the doorways and windows and arrange 3D assets coherently in the scene using constraintbased optimization.”
From Holodeck: Language Guided Generation of 3D Embodied AI Environments (2024) · cites this paper as [37]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“[41] famously found evidence of a ‘cat neuron’, a phenomenon that has been remarkably replicable across newer models and analyses [14,42,43]”
From Symbols and grounding in large language models (2023) · cites this paper as [43]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Text-to-image models like Stable Diffusion, Midjourney, and DALL-E [26,27] enable educators to create visual learning materials from textual content.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [26]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Research works printed but not linked: 24
- Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. A neural probabilistic language model. Journal of machine learning research , 3(Feb):1137-1155, 2003.
- Browne, S. Dark Matters: Surveillance of Blackness . Duke University Press, 2015.
- Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Learning universal imagetext representations. arXiv preprint arXiv:1909.11740 , 2019. · arXiv
- Coates, A., Ng, A., and Lee, H. An analysis of singlelayer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pp. 215-223, 2011.
- Crawford, K. The trouble with bias. NIPS 2017 Keynote , 2017. URL https://www.youtube.com/ watch?v=fMym\_BKWQzk .
- Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: A deep visualsemantic embedding model. In Advances in neural information processing systems , pp. 2121-2129, 2013.
- Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 , 2018. · arXiv
- Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014. · arXiv
- Ha, D., Dai, A., and Le, Q. V. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016. · arXiv
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016b.
- Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019. · arXiv
- Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., and Song, D. Pretrained transformers improve out-ofdistribution robustness. arXiv preprint arXiv:2004.06100 , 2020b. · arXiv
- Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems , pp. 125-136, 2019.
- Jaderberg, M., Simonyan, K., Zisserman, A., et al. Spatial transformer networks. Advances in neural information processing systems , 28:2017-2025, 2015.
- Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 2901-2910, 2017.
- Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. Advances in neural information processing systems , 28: 3294-3302, 2015.
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012.
- Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision , pp. 740-755. Springer, 2014.
- Proceedings of the European Conference on Computer Vision (ECCV) , pp. 181-196, 2018.
- McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. In Advances in neural information processing systems , pp. 6294-6305, 2017.
- Socher, R. and Fei-Fei, L. Connecting modalities: Semisupervised segmentation and annotation of images using unaligned text corpora. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pp. 966-973. IEEE, 2010.
- Srivastava, N. and Salakhutdinov, R. Multimodal learning with deep boltzmann machines. In NIPS , 2012.
- Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.
- Touvron, H., Vedaldi, A., Douze, M., and J´ egou, H. Fixing the train-test resolution discrepancy. In Advances in neural information processing systems , pp. 8252-8262, 2019.
Cited by
Sorted by citations
Items 201–300 of 1,203
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19784-0_6 · Cited by 430
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., & Huang, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01239 · Cited by 431
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
“Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
BackgroundMethodTEMOS: Generating Diverse Human Motions from Textual Descriptions
Petrovich, M., Black, M. J., & Varol, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_28 · Cited by 429
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428
Self-citationTowards Universal Fake Image Detectors that Generalize Across Generative Models
Ojha, U., Li, Y., & Lee, Y. J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02345 · Cited by 428
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_23 · Cited by 427
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
Hu, M., Yin, W., Zhang, C., Cai, Z., Long, X., Chen, H., Wang, K., Yu, G., Shen, C., & Shen, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3444912 · Cited by 423
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models
Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Yang, C., Yang, C., He, Y., Yu, J., Yang, P., et al. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02295-1 · Cited by 436
LViT: Language Meets Vision Transformer in Medical Image Segmentation
Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lü, L., Jin, D., Zhang, Y., & Hong, Q. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3291719 · Cited by 416
LangSplat: 3D Language Gaussian Splatting
Qin, M., Li, W., Zhou, J., Wang, H., & Pfister, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01895 · Cited by 411
On Neural Differential EquationsPreprint
Kidger, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.02435 · Cited by 410
VBench: Comprehensive Benchmark Suite for Video Generative Models
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02060 · Cited by 407
Simple Open-Vocabulary Object Detection
Minderer, M., Gritsenko, A. A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_42 · Cited by 406
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · arXiv · doi:10.48550/arxiv.2104.14548 · Cited by 405
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404
A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., & Mottaghi, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_9 · Cited by 405
Visual Prompt Multi-Modal Tracking
Zhu, J., Lai, S., Chen, X., Wang, D., & Lu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00918 · Cited by 404
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
Wang, C., Chai, M., He, M., Chen, D., & Liao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00381 · Cited by 403
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., & Ling, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19772-7_1 · Cited by 401
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01763 · Cited by 401
Inversion-based Style Transfer with Diffusion Models
Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00978 · Cited by 409
VLMo: Unified Vision-Language Pre-Training with Mixture-Of-Modality-Experts
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., & Wei, F. (2022) · neural information processing systems · doi:10.52202/068431-2384 · Cited by 420
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models
Li, Y., Wang, C., & Jia, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72952-2_19 · Cited by 403
MotionCLIP: Exposing Human Motion Generation to CLIP Space
Tevet, G., Gordon, B., Hertz, A., Bermano, A. H., & Cohen‐Or, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_21 · Cited by 397
OpenScene: 3D Scene Understanding with Open Vocabularies
Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., & Funkhouser, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00085 · Cited by 395
AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models
Gu, Z., Zhu, B., Zhu, G., Chen, Y., Tang, M., & Wang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27963 · Cited by 400
Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., & Li, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01369 · Cited by 393
Image Data Augmentation Approaches: A Comprehensive Survey and Future Directions
Kumar, T., Brennan, R., Mileo, A., & Bendechache, M. (2024) · IEEE Access · doi:10.1109/access.2024.3470122 · Cited by 399
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
Zhang, W., Zhai, G., Wei, Y., Yang, X., & Ma, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01352 · Cited by 392
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training ParadigmPreprint
Li, Y., Liang, F., Zhao, L.-C., Cui, Y., Ouyang, W., Shao, J., Yu, F., & Yan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05208 · Cited by 391
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389
Segment Anything for Microscopy
Archit, A., Freckmann, L., Nair, S., Khalid, N., Hilt, P., Rajashekar, V., Freitag, M., Teuber, C., Spitzner, M., Contreras, C. T., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02580-4 · Cited by 426
FLatten Transformer: Vision Transformer using Focused Linear Attention
Han, D., Pan, X., Han, Y., Song, S., & Huang, G. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00548 · Cited by 386
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation
Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., & Zuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01461 · Cited by 386
A taxonomy of prompt modifiers for text-to-image generation
Oppenlaender, J. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2286532 · Cited by 385
SLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 385
Prompt-aligned Gradient for Prompt Tuning
Zhu, B., Niu, Y., Han, Y., Wu, Y., & Zhang, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01435 · Cited by 387
A Comprehensive Survey on Deep Graph Representation Learning
Ju, W., Fang, Z., Gu, Y., Liu, Z., Long, Q., Qiao, Z., Qin, Y., Shen, J., Sun, F., Xiao, Z., et al. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106207 · Cited by 395
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 382
PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting
Xue, H., & Salim, F. D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3342137 · Cited by 378
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval
Ma, Y., Xu, G., Sun, X., Yan, M., Zhang, J., & Ji, R. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547910 · Cited by 377
Visual Language Maps for Robot Navigation
Huang, C., Mees, O., Zeng, A., & Burgard, W. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160969 · Cited by 376
Class-Incremental Learning: A Survey
Zhou, D.-W., Wang, Q.-W., Qi, Z.-H., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429383 · Cited by 376
One-Step Diffusion with Distribution Matching Distillation
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00632 · Cited by 395
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373
CLIP-Driven Fine-Grained Text-Image Person Re-Identification
Yan, S., Dong, N., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3327924 · Cited by 371
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20215 · Cited by 375
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization
Yang, J., Pavone, M., & Wang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00798 · Cited by 369
Multimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 369
Side Adapter Network for Open-Vocabulary Semantic Segmentation
Xu, M., Zhang, Z., Wei, F., Hu, H., & Bai, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00288 · Cited by 368
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., & Raff, E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_6 · Cited by 367
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Jiang, D., & Ye, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00273 · Cited by 367
CLIP-ReID: Exploiting Vision-Language Model for Image Re-identification without Concrete Text Labels
Li, S., Li, S., & Li, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25225 · Cited by 367
Remote Sensing Object Detection in the Deep Learning Era—A Review
Gui, S., Song, S., Qin, R., & Tang, Y. (2024) · Remote Sensing · doi:10.3390/rs16020327 · Cited by 367
DriveDreamer: Towards Real-World-Drive World Models for Autonomous Driving
Wang, X., Zhu, Z. H., Huang, G., Chen, X., Zhu, J., & Lu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_4 · Cited by 365
Segment anything model for medical image segmentation: Current applications and future directions
Zhang, Y., Shen, Z., & Jiao, R. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108238 · Cited by 364
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363
DocFormer: End-to-End Transformer for Document Understanding
Appalaraju, S., Jasani, B., Kota, B. U., Xie, Y., & Manmatha, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00103 · Cited by 363
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., & Chang, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_2 · Cited by 359
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358
Text2Mesh: Text-Driven Neural Stylization for Meshes
Michel, O., Bar‐On, R., Liu, R., Benaim, S., & Hanocka, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01313 · Cited by 359
High-resolution image reconstruction with latent diffusion models from human brain activity
Takagi, Y., & Nishimoto, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01389 · Cited by 362
Multimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355
Multi-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 353
Uni-Mol: A Universal 3D Molecular Representation Learning FrameworkPreprint
Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L., & Ke, G. (2023) · ChemRxiv · doi:10.26434/chemrxiv-2022-jjm0j-v4 · Cited by 376
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
Li, W., Gao, C., Niu, G., Xiao, X., Liu, H., Liu, J., Wu, H. R., & Wang, H. (2021) · doi:10.18653/v1/2021.acl-long.202 · Cited by 362
Diffusion Model Alignment Using Direct Preference Optimization
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., & Naik, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00786 · Cited by 359
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
Yi, T., Fang, J., Wang, J., Wu, G., Xie, L., Zhang, X., Liu, W., Tian, Q., & Wang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00649 · Cited by 348
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundAvrahami, O., Fried, O., & Lischinski, D. (2023) · ACM Transactions on Graphics · doi:10.1145/3592450 · Cited by 345
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking
Chen, B., Li, P., Bai, L., Qiao, L., Shen, Q., Li, B., Gan, W., Wu, W., & Ouyang, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_22 · Cited by 345
Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
Bianchi, F., Kalluri, P., Durmus, E., Ladhak, F., Cheng, M., Nozza, D., Hashimoto, T., Jurafsky, D., Zou, J., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594095 · Cited by 342
“We use CLIP-L-14 to obtain the image representations (Radford et al., 2021).”
BackgroundMethodBLINK: Multimodal Large Language Models Can See but Not Perceive
Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., & Krishna, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73337-6_9 · Cited by 345
Decoupling Zero-Shot Semantic Segmentation
Ding, J., Xue, N., Xia, G.-S., & Dai, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01129 · Cited by 339
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00779 · Cited by 361
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Metzer, G., Richardson, E., Patashnik, O., Giryes, R., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01218 · Cited by 336
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Xue, L., Gao, M., Chen, X., Martín-Martín, R., Wu, J., Xiong, C., Xu, R., Niebles, J. C., & Savarese, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00120 · Cited by 336
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01742 · Cited by 333
Hong, F., Zhang, M., Pan, L., Cai, Z., Yang, L., & Liu, Z. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530094 · Cited by 335
Audioclip: Extending Clip to Image, Text and Audio
Guzhov, A., Raue, F., Hees, J., & Dengel, A. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747631 · Cited by 330
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 330
Self-citationVL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329
Gaussian Grouping: Segment and Edit Anything in 3D Scenes
Ye, M., Danelljan, M., Yu, F., & Ke, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73397-0_10 · Cited by 329
Generating Human Motion from Textual Descriptions with Discrete Representations
Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., & Ying, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01415 · Cited by 328
“Similarly, MotionCLIP [65] also tends to align text and motion embedding but proposes to utilize CLIP [55] as the text encoder and employ rendered images as extra supervision.”
BackgroundMethodSimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.295 · Cited by 329
On the challenges and perspectives of foundation models for medical image analysis
Zhang, S., & Metaxas, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102996 · Cited by 340
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding
Afham, M., Dissanayake, I., Dissanayake, D., Dharmasiri, A., Thilakarathna, K., & Rodrigo, R. P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00967 · Cited by 327
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
Zhou, G., Hong, Y., & Wu, Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28597 · Cited by 328
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
“Substantial prior work has shown that these models ’ large parameter counts and dataset sizes enable them to effectively model disparate modalities in the same shared latent space, thereby deriving richer representations for each modality than possible with unimodal models 48 – 51 .”
BackgroundTranslation between Molecules and Natural Language
Edwards, C., Lai, T., Ros, K., Honke, G., Cho, K., & Ji, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.26 · Cited by 341
VLP: A Survey on Vision-language Pre-training
Chen, F., Zhang, D., Han, M., Chen, X.-Y., Shi, J. C., Xu, S., & Xu, B. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1369-5 · Cited by 330
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Jayasumana, S., Ramalingam, S., Veit, A., Gläsner, D., Chakrabarti, A., & Kumar, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00889 · Cited by 320
Towards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 318
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317
“A variety of self-supervised VLP approaches have been proposed towards jointly learning visual and textual representations of paired data without supervision, such as frameworks using contrastive objectives [26, 42, 60], approaches based on joint transformer architectures [40, 41, 51, 69], self-supervised VLP with wor…”
BackgroundMethodInfluentialFeature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields
Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z., Xu, D., Chari, P., You, S. O., Wang, Z., & Kadambi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02048 · Cited by 317
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.51 · Cited by 317
Metadata
- Authors
- Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I.
- Affiliations
- OpenAI (United States) (US)
- ORCID
- Kim, Jong Wook
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.00020
- OpenAlex
- W3135367836
- Cited by
- 15,293
- References
- 193
- Citations by year
- 2026: 214 · 2025: 3988 · 2024: 6413 · 2023: 3389 · 2022: 932 · 2021: 349
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- natural language supervision, zero-shot transfer, image representation learning, image captioning, visual concept learning, cross-modal learning, human action recognition, geolocation, optical character recognition, CLIP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
