The DOI matches its DataCite record; 241 of the 241 references it cites were found in a registry or an index; none of the 170 sources its claims cite is checked for support yet.
A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
Open accessAbstract
Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.
Overview
CoChat Summary
TL;DR
This survey systematically reviews prompt engineering for pre-trained vision-language foundation models across three families: multimodal-to-text generation, image-text matching, and text-to-image generation. It organizes prompting methods into hard and soft prompts, surveys applications and responsible-AI issues, and compares prompting on VLMs with prompting on language and vision models.
Key findings
- The paper proposes a taxonomy of prompting methods: hard prompts (task instruction, in-context learning, retrieval-based prompting, chain-of-thought prompting) and soft prompts (prompt tuning, prefix token tuning), while excluding architecture-modifying methods such as P-tuning and LoRA.
- It surveys prompting across three VLM types: multimodal-to-text generation (e.g., Flamingo), image-text matching (e.g., CLIP), and text-to-image generation (e.g., Stable Diffusion), covering applications including VQA, captioning, image classification, object detection, semantic segmentation, and text-to-video/3D/motion generation.
- The survey reports that in-context learning transfers to VLMs but with limits: Frozen performs well in a two-way setting with two new words but fails to transfer to a five-way setting with five new words; increasing in-context samples helps until saturation, and repeated content can hurt performance.
- Responsible-AI concerns are summarized across model types, including adversarial robustness, backdoor attacks, fairness/bias, and privacy; cited examples include a five-character text perturbation shifting Stable Diffusion output and Stable Diffusion generating only white male-appearing firefighters.
Methods
- Design: a comprehensive literature survey that builds a taxonomy and reviews cutting-edge research on prompt engineering for pre-trained VLMs; no original experiments are reported.
- Scope: three VLM categories—multimodal-to-text generation, image-text matching, and text-to-image generation—with prompting methods that maintain the model architecture; methods like P-tuning and LoRA are not the primary scope.
- Analysis: organizes prompting by model type, encoder-side vs decoder-side prompting, hard vs soft prompts, applications, responsible-AI considerations, comparisons with prompting in NLP and pure vision models, and challenges/future directions.
Limitations
- Paper-stated: the rapidly evolving field and wide range of existing prompt engineering approaches make an exhaustive overview challenging; the survey focused primarily on pre-trained VLMs from a prompting perspective and may not cover all recent advancements in other related areas.
- Evident: despite the title “systematic survey,” the text reports no systematic search protocol, databases, inclusion/exclusion criteria, or quantitative synthesis; coverage is narrative and selective.
- Evident: the survey explicitly excludes methods that modify model architecture, such as P-tuning and LoRA, so its scope is narrower than all prompt-engineering work on VLMs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 8 Not valid · 116 Not checked · 42 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One key finding is the versatility and applicability of prompt engineering across different types of vision-language models, including multimodal-to-text generation models, image-textmatching models, and text-to-image generation models.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 8 CONCLUSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Conversely, soft prompts based on image-text matching models, such as CLIP [2], have received extensive investigation.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 7 CHALLENGES AND OPPORTUNITIES
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One of the significant challenges in the field of prompting text-to-other generation models, particularly in the case of Text-to-Video (T2V) and Textto-3D (T2-3D) models, is their dependency on Text-to-Image (T2I) models.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 7 CHALLENGES AND OPPORTUNITIES
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Incontext prompts show surprising results in few-shot [1,16,17,15] and zero-shot scenarios [29,30,1,15].”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > Visual Question Answering.
What the sources say
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [29]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This makes it possible to apply large-scale pre-trained models for realworld applications.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 1 INTRODUCTION
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
55 more claims
Create a free account to see all 166 claims and every source verdict.
- Probably not a substantial claim. 82
- It may state two separate claims as one. 49
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 4.1 Preliminary of Image-Text Matching Models
“Prominent models in this field, such as CLIP [2], ALIGN [58], ALBEF [58] and Multi-Event CLIP [59]”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > References
“58. J. Li et al. Align before fuse: Vision and language representa- tion learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 8 CONCLUSION
“Fig. 2: Prompting methods in multimodal-to-text generation.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 8 CONCLUSION
“Figure 2 illustrates a typical text-image generation framework, highlighting its key components and functionalities”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > Prompting Models with Decoder-based Fusion Module.
“Frozen [44] and BLIP-2 [45] exemplify models that employ image conditional prefix tuning.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > References
“44. D. Noever and S. E. M. Noever. The multimodal and modular ai chef: Complex recipe generation from imagery. arXiv preprint arXiv:2304.02016, 2023.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 4.1 Preliminary of Image-Text Matching Models
“By prompting, we substitute model input by following learnable prompts: for textual prompts: f l v ([{v i } M i=1 , {z i } M i=1 ] ) and for the general visual prompt: f l t ([{c i } M i=1 , {t i } M i=1 ]”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 8 CONCLUSION
“| {z i } M i=1 | visual prompt tokens | | {t i } M i=1 | textual prompt tokens |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > 3.4 Understanding Prompting
“both OFA [49] and KOSMOS-1 [15] add language-only tasks during training to prevent the loss of language ability.”
§ A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models > References
“49. Y. Zang et al. Unified Vision and Language Prompt Learning. arXiv:2210.07225, 2022.”
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“Generative adversarial network (GAN) [104] then used to design end-to-end differentiable image generation structure [105] which is followed by many works [106, 107, 108].”
MethodHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 3×
“However, this survey focuses exclusively on prompt methods that do not involve modifying the underlying model itself, and thus techniques like Ptuning [13] and LoRa [14], which alter the fundamental structure of the model, are not within the primary scope of this study.”
MethodKirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
NeRF: representing scenes as neural radiance fields for view synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2022) · Communications of the ACM · doi:10.1145/3503250 · Cited by 6,852
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637 · cited 1×
“Diffusion models, also known as diffusion probabilistic models [115], originate from non-equilibrium statistical physics [116] and sequential Monte Carlo [117] and are designed to fit any data distribution while keeping tractable.”
BackgroundAn Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · cited 1×
“Besides, variational auto-encoder (VAE) [109] is also adapted to generate images [110, 111].”
MethodGenerative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,796
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00686 · Cited by 1,451
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1×
“In addition, BLIP-2 [32] examines the fusion of two distinct modules for integration: the decoder-based OPT [34] and the encoder-decoder based FlanT5 [35].”
BackgroundGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 2×
“To bypass the common demand of user-defined spatially fixed masks [124, 125], Prompt-to-Prompt [134] can edit images by only editing prompts by replacing a word, specifying a style, changing adjectives, etc .”
BackgroundKosmos-2: Grounding Multimodal Large Language Models to the WorldPreprint
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., & Wei, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14824 · Cited by 1,248
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01832 · Cited by 925
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 4×
“In addition, BLIP-2 [32] examines the fusion of two distinct modules for integration: the decoder-based OPT [34] and the encoder-decoder based FlanT5 [35].”
BackgroundMethodInfluentialLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863
Magic3D: High-Resolution Text-to-3D Content Creation
Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., & Lin, T.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00037 · Cited by 857
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 841
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.15010 · Cited by 837
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Diffusion Models for Adversarial PurificationPreprint
Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., & Anandkumar, A. (2022) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2205.07460 · Cited by 808
Learning to Prompt for Continual Learning
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., & Pfister, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00024 · Cited by 778
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00582 · Cited by 767
Decomposed Prompting: A Modular Approach for Solving Complex TasksPreprint
Khot, T., Trivedi, H. K., Finlayson, M., Fu, Y., Richardson, K., Clark, P. A., & Sabharwal, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.02406 · Cited by 722
Multi-Concept Customization of Text-to-Image Diffusion
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00192 · Cited by 674 · cited 1×
“Custom Diffusion [128] extended the customization into a multi-concept scenario where multiple personalized concepts are composed in the same generated image, such as family members in the same family photo.”
BackgroundOtter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 655
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., & Lu, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01755 · Cited by 631
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024) · AI Open · doi:10.1016/j.aiopen.2023.08.012 · Cited by 596
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., & Shou, M. Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00701 · Cited by 575
Variational Autoencoder for Deep Learning of Images, Labels and Captions
Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., & Carin, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08976 · Cited by 562
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01519 · Cited by 554
On Evaluating Adversarial Robustness of Large Vision-Language Models
Zhao, Y., Pang, T., Du, C., Yang, X., Li, C., Cheung, N. M., & Lin, M. (2023) · neural information processing systems · doi:10.52202/075280-2355 · Cited by 534
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.02643 · Cited by 527
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 512
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · cited 2×
“Language-based VLMs inherit the risks of the underlying LLMs and vision models, such as gender and racial biases when prompted with images [52].”
BackgroundDreamFusion: Text-to-3D using 2D Diffusion
Poole, B., Jain, A., Barron, J. T., & Mildenhall, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14988 · Cited by 474
DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19809-0_36 · Cited by 472
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · cited 1×
“Last but not least, managing the accompanying risks of prompt engineering and large models requires the close collaboration of society, research institutions, and government [214, 215].”
BackgroundReasoning with Language Model Prompting: A SurveyPreprint
Qiao, S.-F., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S.-M., Tan, C.-Q., Huang, F., & Chen, H.-J. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09597 · Cited by 459
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
Complexity-Based Prompting for Multi-Step ReasoningPreprint
Fu, Y., Peng, H. M., Sabharwal, A., Clark, P. A., & Khot, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00720 · Cited by 427
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 424
Diffusion Self-Guidance for Controllable Image Generation
Epstein, D., Jabri, A., Poole, B., Efros, A. A., & Holynski, A. (2023) · neural information processing systems · doi:10.52202/075280-0714 · Cited by 423
Imagen Video: High Definition Video Generation with Diffusion Models
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02303 · Cited by 399
Make-A-Video: Text-to-Video Generation without Text-Video Data
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14792 · Cited by 395
Visual Prompting Via Image Inpainting
Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., & Efros, A. A. (2022) · neural information processing systems · doi:10.52202/068431-1813 · Cited by 394
Prompting Visual-Language Models for Efficient Video Understanding
Ju, C., Han, T., Zheng, K., Zhang, Y., & Xie, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19833-5_7 · Cited by 391
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., & Guo, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00985 · Cited by 389
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511998 · Cited by 389
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image SynthesisPreprint
Feng, W., He, X., Fu, T.-J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X. E., & Wang, W. Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.05032 · Cited by 381
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.10789 · Cited by 371
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Shtedritski, A., Rupprecht, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01101 · Cited by 362
Prompt-to-Prompt Image Editing with Cross Attention Control
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., & Cohen‐Or, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.01626 · Cited by 360
Multimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355
NVAE: A Deep Hierarchical Variational Autoencoder
Vahdat, A., & Kautz, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03898 · Cited by 350
Avrahami, O., Fried, O., & Lischinski, D. (2023) · ACM Transactions on Graphics · doi:10.1145/3592450 · Cited by 345 · cited 2×
“To bypass the common demand of user-defined spatially fixed masks [124, 125], Prompt-to-Prompt [134] can edit images by only editing prompts by replacing a word, specifying a style, changing adjectives, etc .”
BackgroundLearning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · cited 1×
“Unlike hard prompts, soft prompts are characterized as continuous vectors that can be fine-tuned using gradient-based meth-ods [6, 41].”
MethodA generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 327
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20215 · Cited by 326
FLAME: Free-Form Language-Based Motion Synthesis & Editing
Kim, J.-H., Kim, J., & Choi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.25996 · Cited by 314
Template-Based Named Entity Recognition Using BART
Cui, L., Wu, Y., Liu, J., Yang, S., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.161 · Cited by 314
SegGPT: Towards Segmenting Everything In Context
Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., & Huang, T. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00110 · Cited by 310
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 1×
“Press et al. [188] examine the capacity of language models to execute compositional reasoning tasks and found that LLMs are good at memorizing facts but do not compose them to answer questions.”
BackgroundTevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., & Bermano, A. H. (2022) · arXiv · doi:10.48550/arxiv.2209.14916 · Cited by 290
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · arXiv · doi:10.48550/arxiv.2101.00190 · Cited by 289
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13921 · Cited by 273
Social Biases through the Text-to-Image Generation Lens
Naik, R., & Nushi, B. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604711 · Cited by 272
Scaling Robot Learning with Semantically Imagined Experience
Yu, T., Xiao, T. G., Tompson, J., Stone, A., Wang, S., Brohan, A., Singh, J., Tan, C., Dee, M., Peralta, J., et al. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.027 · Cited by 271
DiffRF: Rendering-Guided 3D Radiance Field Diffusion
Muller, N. E., Siddiqui, Y., Porzi, L., Bulò, S. R., Kontschieder, P., & Nießner, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00421 · Cited by 263
Unified Vision and Language Prompt LearningPreprint
Zang, Y., Li, W., Zhou, K., Huang, C., & Loy, C. C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07225 · Cited by 259
Stable Bias: Analyzing Societal Representations in Diffusion ModelsPreprint
Luccioni, A. S., Akiki, C., Mitchell, M., & Jernite, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.11408 · Cited by 257
Wang, P., An, Y., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., & Yang, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03052 · Cited by 250
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00234 · Cited by 248
Unified Demonstration Retriever for In-Context Learning
Li, X., Lv, K., Yan, H., Lin, T., Zhu, W., Ni, Y., Xie, G., Wang, X., & Qiu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.256 · Cited by 243
Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
Xu, J., Wang, X., Cheng, W., Cao, Y., Shan, Y., Qie, X., & Gao, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02003 · Cited by 236
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing
Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., & Chen, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01460 · Cited by 224
Diffusion Models for Imperceptible and Transferable Adversarial Attack
Chen, J., Chen, H., Chen, K., Zhang, Y., Zou, Z., & Shi, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3480519 · Cited by 220
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., & Grave, E. (2022) · arXiv · doi:10.48550/arxiv.2208.03299 · Cited by 213
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
Pan, X., Tewari, A., Leimkühler, T., Liu, L., Meka, A., & Theobalt, C. (2023) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings · doi:10.1145/3588432.3591500 · Cited by 205
DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations
Sun, X., Hu, P., & Saenko, K. (2022) · neural information processing systems · doi:10.52202/068431-2216 · Cited by 204
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · arXiv · doi:10.48550/arxiv.2302.05543 · Cited by 201
Domain Adaptation via Prompt Learning
Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., & Huang, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3327962 · Cited by 199
Fair Diffusion: Instructing Text-to-Image Generation Models on FairnessPreprint
Friedrich, F., Brack, M., Struppek, L., Hintersdorf, D., Schramowski, P., Luccioni, A. S., & Kersting, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.10893 · Cited by 197
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
Wang, X., Wang, W., Cao, Y., Shen, C., & Huang, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00660 · Cited by 197
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · neural information processing systems · doi:10.52202/075280-3155 · Cited by 191
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models
Mohammad Khalid, N., Xie, T., Belilovsky, E., & Popa, T. (2022) · SIGGRAPH Asia 2022 Conference Papers · doi:10.1145/3550469.3555392 · Cited by 190
Debiasing Vision-Language Models via Biased PromptsPreprint
Chuang, C.-Y., Jampani, V., Li, Y., Torralba, A. B., & Jegelka, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00070 · Cited by 188
Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream ImplicationsPreprint
Agarwal, S., Krueger, G., Clark, J., Radford, A., Kim, J. W., & Brundage, M. (2021) · arXiv.org · doi:10.48550/arxiv.2108.02818 · Cited by 185
OpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.10 · Cited by 185
Unsupervised Prompt Learning for Vision-Language ModelsPreprint
Huang, T. J., Chu, J., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03649 · Cited by 179
Unifying Vision-and-Language Tasks via Text Generation
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 179
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177 · cited 1×
“Prompt engineering has been widely adopted in various natural language processing applications including question answering [174, 175], text classification [61, 176], text generation [36, 23, 177], and information extraction [178, 179], etc .”
BackgroundPoisoning and Backdooring Contrastive LearningPreprint
Carlini, N., & Terzis, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09667 · Cited by 176
Backdoor Defense via Decoupling the Training ProcessPreprint
Huang, K., Li, Y., Wu, B., Qin, Z., & Ren, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03423 · Cited by 175
Learning from Experiments when Context Matters
Pritchett, L. H., & Sandefur, J. (2015) · American Economic Review · doi:10.1257/aer.p20151016 · Cited by 174
LAS-AT: Adversarial Training with Learnable Attack Strategy
Jia, X., Zhang, Y., Wu, B., Ma, K., Wang, J., & Cao, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01304 · Cited by 171
Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)Preprint
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., & Schmidt, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01397 · Cited by 169
CPT: Colorful Prompt Tuning for pre-trained vision-language models
Yao, Y., Zhang, A., Zhang, Z., Liu, Z., Chua, T.-S., & Sun, M. (2024) · AI Open · doi:10.1016/j.aiopen.2024.01.004 · Cited by 167
PromptDet: Towards Open-Vocabulary Detection Using Uncurated Images
Feng, C., Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X., Xie, W., & Ma, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_41 · Cited by 161
Wu, W., Zhao, Y., Shou, M. Z., Zhou, H., & Shen, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00117 · Cited by 160
BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning
Jia, J., Liu, Y., & Gong, N. Z. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833644 · Cited by 158
Zhang, S., Khan, S. A., Shen, Z., Naseer, M., Chen, G., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00339 · Cited by 157
Understanding Zero-Shot Adversarial Robustness for Large-Scale ModelsPreprint
Mao, C., Geng, S., Yang, J., Wang, X. E., & Vondrick, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.07016 · Cited by 149
What Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148
Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning
Zhai, S., Dong, Y., Shen, Q., Pu, S., Fang, Y., & Su, H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612108 · Cited by 147
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
Kazemi, M., Kim, N., Bhatia, D. S., Xu, X., & Ramachandran, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.361 · Cited by 142
Diversity-Aware Meta Visual Prompting
Huang, Q., Dong, X., Chen, D., Zhang, W., Wang, F., Hua, G., & Yu, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01047 · Cited by 131
Exploring Visual Prompts for Adapting Large-Scale Models
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022) · arXiv · doi:10.48550/arxiv.2203.17274 · Cited by 129
Visual Prompt Tuning for Test-time Domain AdaptationPreprint
Gao, Y., Shi, X., Zhu, Y., Wang, H., Tang, Z., Zhou, X., Li, M., & Metaxas, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.04831 · Cited by 129
Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis
Struppek, L., Hintersdorf, D., & Kersting, K. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00423 · Cited by 129
The Turking Test: Can Language Models Understand Instructions?Preprint
Efrat, A., & Levy, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11982 · Cited by 128
Multitask Vision-Language Prompt Tuning
Shen, S., Yang, S., Zhang, T., Zhai, B., Gonzalez, J. E., Keutzer, K., & Darrell, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00556 · Cited by 128
Compositional Exemplars for In-context LearningPreprint
Ye, J., Wu, Z., Feng, J., Yu, T., & Kong, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05698 · Cited by 125
Shabani, M. A., Hosseini, S., & Furukawa, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00529 · Cited by 124 · cited 1×
“that represent individual rooms to generate house floorplans, facilitating the design and planning of residential spaces [154].”
BackgroundBackdoor Defense via Adaptively Splitting Poisoned Dataset
Gao, K., Bai, Y., Gu, J., Yang, Y., & Xia, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00390 · Cited by 119
Self-citationZhang, K., Yu, J., Yan, Z., Liu, Y., Adhikarla, E., Fu, S., Chen, X., Chen, C., Zhou, Y., Li, X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.17100 · Cited by 116
CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning
Bansal, H., Yin, F., Singhi, N., Grover, A., Yu, Y., & Chang, K.-W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00017 · Cited by 114
Are Vision Transformers Robust to Patch Perturbations?
Gu, J., Tresp, V., & Qin, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_24 · Cited by 111
Self-citationFew-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107 · cited 1×
“Prompt engineering has been widely adopted in various natural language processing applications including question answering [174, 175], text classification [61, 176], text generation [36, 23, 177], and information extraction [178, 179], etc .”
BackgroundExploring the Universal Vulnerability of Prompt-based Learning Paradigm
Xu, L., Chen, Y., Cui, G., Gao, H., & Liu, Z. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.137 · Cited by 104
Towards Open-Vocabulary Scene Graph Generation with Prompt-Based Finetuning
He, T., Gao, L., Song, J., & Li, Y.-F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_4 · Cited by 97
Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
Rempe, D., Luo, Z., Peng, X. B., Yuan, Y., Kitani, K., Kreis, K., Fidler, S., & Litany, O. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01322 · Cited by 97 · cited 1×
“In autonomous driving, diffusion models are leveraged to generate controllable pedestrian trajectories that align with the surrounding environment’s context that enables the simulation of realistic pedestrian behavior [153].”
BackgroundMembership Inference Attacks Against Text-to-image Generation ModelsPreprint
Wu, Y., Ning, Y., Li, Z., Backes, M., & Zhang, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.00968 · Cited by 95
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Paranjape, B., Michael, J., Ghazvininejad, M., Hajishirzi, H., & Zettlemoyer, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.366 · Cited by 93
Are Diffusion Models Vulnerable to Membership Inference Attacks?Preprint
Duan, J., Kong, F., Wang, S., Shi, X., & Xu, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.01316 · Cited by 90
Tu, C.-H., Mai, Z., & Chao, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00746 · Cited by 89
Discovering and Mitigating Visual Biases Through Keyword Explanation
Kim, Y., Mo, S., Kim, M., Lee, K., Lee, J., & Shin, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01054 · Cited by 87
Vision-and-Language Pretrained Models: A Survey
Long, S., Cao, F., Han, S. C., & Yang, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/773 · Cited by 87
Otter: A Multi-Modal Model with In-Context Instruction TuningPreprint
Li, B., Zhang, Y., Chen, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03726 · Cited by 86
Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models
Huang, Y., Juefei-Xu, F., Guo, Q., Zhang, J., Wu, Y., Hu, M., Li, T., Pu, G., & Liu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30110 · Cited by 85
Investigating Prompt Engineering in Diffusion ModelsPreprint
Witteveen, S., & Andrews, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.15462 · Cited by 83
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Paranjape, B., Michael, J., Ghazvininejad, M., Zettlemoyer, L., & Hajishirzi, H. (2021) · Findings · doi:10.48550/arxiv.2106.06823 · Cited by 81
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language ModelsPreprint
Yao, Y., Zhang, A., Zhang, Z., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11797 · Cited by 80
A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion
Zhuang, H., Zhang, Y., & Liu, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00236 · Cited by 78
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O'Keefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Cass-Beggs, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03718 · Cited by 77
Zhu, J., Yang, H., He, H., Wang, W., Tuo, Z., Cheng, W.-H., Gao, L., Song, J., & Fu, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612707 · Cited by 77
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
Wang, J., Liu, X., Di, Z., Liu, Y., & Wang, X. E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.160 · Cited by 76
Prompt Vision Transformer for Domain GeneralizationPreprint
Zheng, Z., Yue, X., Wang, K., & You, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.08914 · Cited by 75
A Reproducible Extraction of Training Images from Diffusion ModelsPreprint
Webster, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08694 · Cited by 70
SegGPT: Segmenting Everything In ContextPreprint
Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., & Huang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.03284 · Cited by 70
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
Wang, J.-L., Liu, X., Di, Z.-L., Liu, Y., & Wang, X. E. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2306.00905 · Cited by 62
Towards Efficient Adversarial Training on Vision Transformers
Wu, B., Gu, J., Li, Z., Cai, D., He, X., & Liu, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19778-9_18 · Cited by 59
Self-citationUnderstanding Pure CLIP Guidance for Voxel Grid NeRF ModelsPreprint
Lee, H.-H., & Chang, A. X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15172 · Cited by 58
Delobelle, P., Tokpo, E., Calders, T., & Berendt, B. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.122 · Cited by 57
Unleashing the Power of Visual Prompting At the Pixel Level
Wu, J., Li, X., Wei, C., Wang, H., Yuille, A., Zhou, Y., & Xie, C. (2022) · Transactions on machine learning research · doi:10.48550/arxiv.2212.10556 · Cited by 55
LPT: Long-tailed Prompt Tuning for Image ClassificationPreprint
Dong, B., Zhou, P., Yan, S., & Zuo, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.01033 · Cited by 54
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
Chen, S., Gu, J., Han, Z., Ma, Y., Torr, P. H. S., & Tresp, V. (2023) · neural information processing systems · doi:10.52202/075280-2255 · Cited by 51
Self-citationMAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning
Eichenberg, C., Black, S., Weinbach, S., Parcalabescu, L., & Frank, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.179 · Cited by 49
A prompt-based approach to adversarial example generation and robustness enhancement
Yang, Y., Huang, P., Cao, J., Li, J., Lin, Y., & Ma, F. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-2639-2 · Cited by 46
Multimodal Procedural Planning via Dual Text-Image PromptingPreprint
Lu, Y., Lu, P., Chen, Z., Zhu, W., Wang, X. E., & Wang, W. Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.01795 · Cited by 44
Prompt Tuning for Generative Multimodal Pretrained ModelsPreprint
Yang, H., Lin, J., An, Y., Wang, P., Zhou, C., & Yang, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.02532 · Cited by 43
Is Prompt All You Need? No. A Comprehensive and Broader View of Instruction LearningPreprint
Lou, R., Zhang, K., & Yin, W. (2023) · arXiv.org · doi:10.48550/arxiv.2303.10475 · Cited by 42
Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion ModelsPreprint
Jeong, H., Kwon, G., & Ye, J. C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.03900 · Cited by 37
Prompt Generation Networks for Input-Space Adaptation of Frozen Vision TransformersPreprint
Loedeman, J., Stol, M., Han, T., & Asano, Y. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06466 · Cited by 36
Visual Prompting for Adversarial Robustness
Chen, A., Lorenz, P., Yao, Y., Chen, P., & Liu, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097245 · Cited by 36
Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast SetsPreprint
Smith, B. M., Farinha, M., Hall, S. M., Kirk, H. R., Shtedritski, A., & Bain, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15407 · Cited by 34
DeAR: Debiasing Vision-Language Models with Additive Residuals
Seth, A., Hemani, M., & Agarwal, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00659 · Cited by 34
Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation DetectionPreprint
Gao, K., Chen, L., Zhang, H., Xiao, J., & Sun, Q. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00268 · Cited by 31
Multi-event Video-Text Retrieval
Zhang, G., Ren, J., Gu, J., & Tresp, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02021 · Cited by 30
Self-citationThreats to Pre-trained Language Models: Survey and TaxonomyPreprint
Guo, S., Xie, C., Li, J., Lyu, L., & Zhang, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06862 · Cited by 29
Prompt Stealing Attacks Against Text-to-Image Generation ModelsPreprint
Shen, X., Qu, Y., Michael Backes, A., & Zhang, Y. (2023) · arXiv (Cornell University) · doi:10.22028/d291-47438 · Cited by 28
Exploring the Benefits of Visual Prompting in Differential Privacy
Li, Y., Tsai, Y.-L., Yu, C.-M., Chen, P., & Ren, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00476 · Cited by 28
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video GenerationPreprint
Wu, J. Z., Ge, Y., Wang, X., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.11565 · Cited by 27
Magic3D: High-Resolution Text-to-3D Content CreationPreprint
Lin, C.-H., Gao, J., Tang, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.10440 · Cited by 25
Effective Robustness against Natural Distribution Shifts for Models with Different Training Data
Shi, Z., Carlini, N., Balashankar, A., Schmidt, L., Hsieh, C.-J., Beutel, A., & Qin, Y. (2023) · neural information processing systems · doi:10.52202/075280-3217 · Cited by 19
Self-citationThe Multimodal And Modular Ai Chef: Complex Recipe Generation From ImageryPreprint
Noever, D. A., & Noever, S. E. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02016 · Cited by 18
Mitigating Test-Time Bias for Fair Image Retrieval
Kong, F., Yuan, S., Hao, W., & Henao, R. (2023) · neural information processing systems · doi:10.52202/075280-3131 · Cited by 17
Do DALL-E and Flamingo Understand Each Other?
Li, H., Gu, J., Koner, R., Sharifzadeh, S., & Tresp, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00191 · Cited by 17
Self-citation
Research works printed but not linked: 22
- Z. Zhang et al. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
- X. Chen et al. PaLI: A jointly-scaled multilingual language- image model. In The Eleventh International Conference on Learning Representations, 2023.
- A. Radford et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- J. Li et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. · arXiv
- J. B. Nici. AP Art History: 5 Practice Tests + Comprehensive Review + Online Practice. Barron's Educational Series, 2020.
- J. Qiu et al. Benchmarking robustness under distribution shift of multimodal image-text models. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
- J. Li et al. Align before fuse: Vision and language representa- tion learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
- T. Gao et al. Making Pre-trained Language Models Better Few- shot Learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021. Association for Computational Linguistics.
- Z. Guo et al. Texts as images in prompt tuning for multi- label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2808-2817, 2023.
- J. Wen et al. Visual Prompt Tuning for Few-Shot Text Classification. In Proceedings of the 29th International Confer- ence on Computational Linguistics, pp. 5560-5570, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
- Y. Du et al. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
- D. Jeulin. Dead leaves models: from space tesselation to random functions proc. of the symposium on the advances in the theory and applications of random sets (fontainebleau, 9-11 october 1996) ed d jeulin, 1997.
- M. Ni et al. Imaginarynet: Learning object detectors without real images and annotations. In ICLR, 2023.
- R. He et al. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
- R. Gal et al. An image is worth one word: Personalizing text- to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
- J. Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, pp. 679-698, 1986.
- J. Ho et al. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
- P. Lewis et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
- S. Borgeaud et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
- H. Salman et al. Unadversarial examples: Designing objects for robust vision. Advances in Neural Information Processing Systems, 34:15270-15284, 2021.
- A. Madry et al. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2017.
- J. Gu et al. Effective and efficient vote attack on capsule networks. In International Conference on Learning Represen- tations (ICLR), 2021.
Cited by
Unleashing the potential of prompt engineering for large language models
Chen, B., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 490
Generative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333
“Example : The most prominent example of AI legislation to date is the European AI Act [32], and European users should become familiar with it and its implications.”
Vision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 229
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 281
Vision Language Models in Autonomous Driving: A Survey and Outlook
Zhou, X., Liu, M., Yurtsever, E., Žagar, B. L., Zimmer, W. M. X., Cao, H., & Knoll, A. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3402136 · Cited by 401
Metadata
- Authors
- Gu, J., Han, Z., Chen, S., Beirami, A., He, B., Zhang, G., Liao, R., Qin, Y., Tresp, V., & Torr, P.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.12980
- OpenAlex
- W4385262478
- Cited by
- 280
- References
- 107
- Citations by year
- 2026: 60 · 2025: 112 · 2024: 91 · 2023: 17
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- University of Oxford
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- prompt engineering, vision-language foundation models, image-text matching, text-to-image generation, CLIP, flamingos, Stable Diffusion, prompting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.