CoChat

The DOI matches its DataCite record; 241 of the 241 references it cites were found in a registry or an index; none of the 170 sources its claims cite is checked for support yet.

A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

Open access
Gu, Jindong, Han, Zhen, Chen, Shuo, Beirami, Ahmad, He, Bailan, Zhang, Gengyuan, Liao, Ruotong, Qin, Yao, Tresp, Volker, Torr, Philip · 2023 · arXiv

Abstract

Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.

Overview

CoChat Summary

TL;DR

This survey systematically reviews prompt engineering for pre-trained vision-language foundation models across three families: multimodal-to-text generation, image-text matching, and text-to-image generation. It organizes prompting methods into hard and soft prompts, surveys applications and responsible-AI issues, and compares prompting on VLMs with prompting on language and vision models.

Key findings

  • The paper proposes a taxonomy of prompting methods: hard prompts (task instruction, in-context learning, retrieval-based prompting, chain-of-thought prompting) and soft prompts (prompt tuning, prefix token tuning), while excluding architecture-modifying methods such as P-tuning and LoRA.
  • It surveys prompting across three VLM types: multimodal-to-text generation (e.g., Flamingo), image-text matching (e.g., CLIP), and text-to-image generation (e.g., Stable Diffusion), covering applications including VQA, captioning, image classification, object detection, semantic segmentation, and text-to-video/3D/motion generation.
  • The survey reports that in-context learning transfers to VLMs but with limits: Frozen performs well in a two-way setting with two new words but fails to transfer to a five-way setting with five new words; increasing in-context samples helps until saturation, and repeated content can hurt performance.
  • Responsible-AI concerns are summarized across model types, including adversarial robustness, backdoor attacks, fairness/bias, and privacy; cited examples include a five-character text perturbation shifting Stable Diffusion output and Stable Diffusion generating only white male-appearing firefighters.

Methods

  • Design: a comprehensive literature survey that builds a taxonomy and reviews cutting-edge research on prompt engineering for pre-trained VLMs; no original experiments are reported.
  • Scope: three VLM categories—multimodal-to-text generation, image-text matching, and text-to-image generation—with prompting methods that maintain the model architecture; methods like P-tuning and LoRA are not the primary scope.
  • Analysis: organizes prompting by model type, encoder-side vs decoder-side prompting, hard vs soft prompts, applications, responsible-AI considerations, comparisons with prompting in NLP and pure vision models, and challenges/future directions.

Limitations

  • Paper-stated: the rapidly evolving field and wide range of existing prompt engineering approaches make an exhaustive overview challenging; the survey focused primarily on pre-trained VLMs from a prompting perspective and may not cover all recent advancements in other related areas.
  • Evident: despite the title “systematic survey,” the text reports no systematic search protocol, databases, inclusion/exclusion criteria, or quantitative synthesis; coverage is narrative and selective.
  • Evident: the survey explicitly excludes methods that modify model architecture, such as P-tuning and LoRA, so its scope is narrower than all prompt-engineering work on VLMs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.12980

Cites

241 of 241 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 8 Not valid · 116 Not checked · 42 Own results and interpretations

55 more claims

Create a free account to see all 166 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 82
  • It may state two separate claims as one. 49

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Items 201–241 of 241

Research works printed but not linked: 22

  1. Z. Zhang et al. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
  2. X. Chen et al. PaLI: A jointly-scaled multilingual language- image model. In The Eleventh International Conference on Learning Representations, 2023.
  3. A. Radford et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  4. J. Li et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. · arXiv
  5. J. B. Nici. AP Art History: 5 Practice Tests + Comprehensive Review + Online Practice. Barron's Educational Series, 2020.
  6. J. Qiu et al. Benchmarking robustness under distribution shift of multimodal image-text models. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
  7. J. Li et al. Align before fuse: Vision and language representa- tion learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
  8. T. Gao et al. Making Pre-trained Language Models Better Few- shot Learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021. Association for Computational Linguistics.
  9. Z. Guo et al. Texts as images in prompt tuning for multi- label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2808-2817, 2023.
  10. J. Wen et al. Visual Prompt Tuning for Few-Shot Text Classification. In Proceedings of the 29th International Confer- ence on Computational Linguistics, pp. 5560-5570, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
  11. Y. Du et al. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
  12. D. Jeulin. Dead leaves models: from space tesselation to random functions proc. of the symposium on the advances in the theory and applications of random sets (fontainebleau, 9-11 october 1996) ed d jeulin, 1997.
  13. M. Ni et al. Imaginarynet: Learning object detectors without real images and annotations. In ICLR, 2023.
  14. R. He et al. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
  15. R. Gal et al. An image is worth one word: Personalizing text- to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
  16. J. Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, pp. 679-698, 1986.
  17. J. Ho et al. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
  18. P. Lewis et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
  19. S. Borgeaud et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
  20. H. Salman et al. Unadversarial examples: Designing objects for robust vision. Advances in Neural Information Processing Systems, 34:15270-15284, 2021.
  21. A. Madry et al. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2017.
  22. J. Gu et al. Effective and efficient vote attack on capsule networks. In International Conference on Learning Represen- tations (ICLR), 2021.

Cited by

See all 281

Metadata

Authors
Gu, J., Han, Z., Chen, S., Beirami, A., He, B., Zhang, G., Liao, R., Qin, Y., Tresp, V., & Torr, P.
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4385262478
Cited by
280
References
107
Citations by year
2026: 60 · 2025: 112 · 2024: 91 · 2023: 17
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
University of Oxford
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
prompt engineering, vision-language foundation models, image-text matching, text-to-image generation, CLIP, flamingos, Stable Diffusion, prompting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.