Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 281Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningGenerative Adversarial Networks and Image Synthesis

The DOI matches its DataCite record; 195 of the 195 references it cites were found in a registry or an index; none of the 170 sources its claims cite is checked for support yet.

A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

Open access
Gu, Jindong, Han, Zhen, Chen, Shuo, Beirami, Ahmad, He, Bailan, Zhang, Gengyuan, Liao, Ruotong, Qin, Yao, Tresp, Volker, Torr, Philip·2023·arXiv
DOI 10.48550/arxiv.2307.12980arXiv 2307.12980
Open this paper in CoChat — free accountOpen PDF

Abstract

Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.

Overview

CoChat Summary

TL;DR

This survey systematically reviews prompt engineering for pre-trained vision-language foundation models across three families: multimodal-to-text generation, image-text matching, and text-to-image generation. It organizes prompting methods into hard and soft prompts, surveys applications and responsible-AI issues, and compares prompting on VLMs with prompting on language and vision models.

Key findings

  • The paper proposes a taxonomy of prompting methods: hard prompts (task instruction, in-context learning, retrieval-based prompting, chain-of-thought prompting) and soft prompts (prompt tuning, prefix token tuning), while excluding architecture-modifying methods such as P-tuning and LoRA.
  • It surveys prompting across three VLM types: multimodal-to-text generation (e.g., Flamingo), image-text matching (e.g., CLIP), and text-to-image generation (e.g., Stable Diffusion), covering applications including VQA, captioning, image classification, object detection, semantic segmentation, and text-to-video/3D/motion generation.
  • The survey reports that in-context learning transfers to VLMs but with limits: Frozen performs well in a two-way setting with two new words but fails to transfer to a five-way setting with five new words; increasing in-context samples helps until saturation, and repeated content can hurt performance.
  • Responsible-AI concerns are summarized across model types, including adversarial robustness, backdoor attacks, fairness/bias, and privacy; cited examples include a five-character text perturbation shifting Stable Diffusion output and Stable Diffusion generating only white male-appearing firefighters.

Methods

  • Design: a comprehensive literature survey that builds a taxonomy and reviews cutting-edge research on prompt engineering for pre-trained VLMs; no original experiments are reported.
  • Scope: three VLM categories—multimodal-to-text generation, image-text matching, and text-to-image generation—with prompting methods that maintain the model architecture; methods like P-tuning and LoRA are not the primary scope.
  • Analysis: organizes prompting by model type, encoder-side vs decoder-side prompting, hard vs soft prompts, applications, responsible-AI considerations, comparisons with prompting in NLP and pure vision models, and challenges/future directions.

Limitations

  • Paper-stated: the rapidly evolving field and wide range of existing prompt engineering approaches make an exhaustive overview challenging; the survey focused primarily on pre-trained VLMs from a prompting perspective and may not cover all recent advancements in other related areas.
  • Evident: despite the title “systematic survey,” the text reports no systematic search protocol, databases, inclusion/exclusion criteria, or quantitative synthesis; coverage is narrative and selective.
  • Evident: the survey explicitly excludes methods that modify model architecture, such as P-tuning and LoRA, so its scope is narrower than all prompt-engineering work on VLMs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.12980

Cites

195 of 195 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 8 Not valid · 116 Not checked · 42 Own results and interpretations

55 more claims

Create a free account to see all 166 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 82
  • It may state two separate claims as one. 49

Create a free account to see these claims.

Create a free account to see these assumptions.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 107 references

Research works printed but not linked: 67

  1. Z. Zhang et al. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
  2. M. Tsimpoukelli et al. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
  3. C. Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020.
  4. T. Brown et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
  5. J. Lu et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
  6. J. Cho et al. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pp. 1931-1942. PMLR, 2021.
  7. S. Ren et al. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
  8. P. Wang et al. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pp. 23318- 23340. PMLR, 2022.
  9. Z. Wang et al. SimVLM: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations, 2022.
  10. X. Chen et al. PaLI: A jointly-scaled multilingual language- image model. In The Eleventh International Conference on Learning Representations, 2023.
  11. A. Radford et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  12. J. Ye et al. Compositional exemplars for in-context learning. In International Conference on Machine Learning. PMLR, 2023.
  13. J. Li et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. · arXiv
  14. J. B. Nici. AP Art History: 5 Practice Tests + Comprehensive Review + Online Practice. Barron's Educational Series, 2020.
  15. J. Qiu et al. Benchmarking robustness under distribution shift of multimodal image-text models. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
  16. J. Li et al. Align before fuse: Vision and language representa- tion learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
  17. T. Gao et al. Making Pre-trained Language Models Better Few- shot Learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021. Association for Computational Linguistics.
  18. Z. Guo et al. Texts as images in prompt tuning for multi- label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2808-2817, 2023.
  19. J. Wen et al. Visual Prompt Tuning for Few-Shot Text Classification. In Proceedings of the 29th International Confer- ence on Computational Linguistics, pp. 5560-5570, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
  20. Y. Du et al. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
  21. A. Fang et al. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pp. 6216-6234. PMLR, 2022.
  22. K. Gregor et al. Draw: A recurrent neural network for image generation. In International conference on machine learning, pp. 1462-1471. PMLR, 2015.
  23. S. Reed et al. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
  24. P. Isola et al. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
  25. D. P. Kingma et al. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4):307- 392, 2019.
  26. Y. Pu et al. Variational autoencoder for deep learning of images, labels and captions. Advances in neural information processing systems, 29, 2016.
  27. A. Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing sys- tems, 33:19667-19679, 2020.
  28. A. Ramesh et al. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821- 8831. PMLR, 2021.
  29. P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
  30. J. Sohl-Dickstein et al. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
  31. D. Jeulin. Dead leaves models: from space tesselation to random functions proc. of the symposium on the advances in the theory and applications of random sets (fontainebleau, 9-11 october 1996) ed d jeulin, 1997.
  32. J. Ho et al. Denoising diffusion probabilistic models. Ad- vances in Neural Information Processing Systems, 33:6840- 6851, 2020.
  33. M. Ni et al. Imaginarynet: Learning object detectors without real images and annotations. In ICLR, 2023.
  34. R. He et al. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
  35. R. Gal et al. An image is worth one word: Personalizing text- to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
  36. N. Ruiz et al. Dreambooth: Fine tuning text-to-image diffu- sion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
  37. B. Kawar et al. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
  38. J. Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, pp. 679-698, 1986.
  39. J. Ho et al. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
  40. L. Ruan et al. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10219-10228, 2023.
  41. N. Müller et al. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4328-4338, 2023.
  42. B. Mildenhall et al. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
  43. D. Rempe et al. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  44. M. A. Shabani et al. Housediffusion: Vector floorplan gen- eration via a diffusion model with discrete and continuous denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5466-5475, 2023.
  45. A. Jobin et al. The global landscape of ai ethics guidelines. Nature Machine Intelligence, 1(9):389-399, 2019.
  46. A. Seth et al. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6820-6829, 2023.
  47. D. Khashabi et al. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, November 2020. Association for Computational Lin- guistics.
  48. Z. Jiang et al. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021.
  49. T. Schick and H. Schütze. Few-shot text generation with natural language instructions. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pp. 390-402, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  50. X. Chen et al. Knowprompt: Knowledge-aware prompt- tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference 2022, WWW '22, pp. 2778-2788, New York, NY, USA, 2022. Association for Computing Machinery.
  51. L. Cui et al. Template-based named entity recognition using BART. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pp. 1835-1845, Online, August 2021. Association for Computational Linguistics.
  52. L. Ouyang et al. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
  53. P. Lewis et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
  54. S. Borgeaud et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
  55. P. Delobelle et al. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 1693-1706, Seat- tle, United States, July 2022. Association for Computational Linguistics.
  56. N. Ding et al. Openprompt: An open-source framework for prompt-learning. In V. Basile et al., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics, ACL 2022 -System Demonstrations, Dublin, Ireland, May 22-27, 2022, pp. 105-113. Association for Computational Linguistics, 2022.
  57. X. Wang et al. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023.
  58. H. Salman et al. Unadversarial examples: Designing objects for robust vision. Advances in Neural Information Processing Systems, 34:15270-15284, 2021.
  59. R. Girdhar et al. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15180- 15190, June 2023.
  60. K. Gao et al. Backdoor defense via adaptively splitting poisoned dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4005-4014, 2023.
  61. S. Yang et al. Backdoor defense via suppressing model short- cuts. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
  62. A. Madry et al. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2017.
  63. X. Jia et al. Las-at: adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13398-13408, 2022.
  64. B. Wu et al. Towards efficient adversarial training on vision transformers. In European Conference on Computer Vision, pp. 307-325. Springer, 2022.
  65. J. Gu et al. Are vision transformers robust to patch pertur- bations? In European Conference on Computer Vision, pp. 404-421. Springer, 2022.
  66. J. Gu et al. Effective and efficient vote attack on capsule networks. In International Conference on Learning Represen- tations (ICLR), 2021.
  67. H. Li et al. Do dall-e and flamingo understand each other? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (to appear), 2023.

Cited by

See all 281

Related

See all

Metadata

Authors
Gu, J., Han, Z., Chen, S., Beirami, A., He, B., Zhang, G., Liao, R., Qin, Y., Tresp, V., & Torr, P.
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.12980
arXiv
2307.12980
OpenAlex
W4385262478
Cited by
280
References
107
Citations by year
2026: 60 · 2025: 112 · 2024: 91 · 2023: 17
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
University of Oxford
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
prompt engineering, vision-language foundation models, image-text matching, text-to-image generation, CLIP, flamingos, Stable Diffusion, prompting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models. https://app.cochat.ai/papers/10.48550/arxiv.2307.12980
The paper
Gu, J., Han, Z., Chen, S., Beirami, A., He, B., Zhang, G., Liao, R., Qin, Y., Tresp, V., & Torr, P. (2023). A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models. arXiv. https://doi.org/10.48550/arxiv.2307.12980

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.