CoChat
Sign Up
Cited by 1,051Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Neural Network Applications

Vision-Language Models for Vision Tasks: A Survey

Open access
Zhang, Jingyi, Huang, Jiaxing, Jin, Sheng, Lu, Shijian·2024·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2024.3369699
Open this paper in CoChat — free accountOpen PDF

Abstract

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3369699

Cites

192 of 192 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent approaches leverage deep learning models such as transformers and vision-language models to improve detection capabilities.

    “With the rise of multi-modal forgeries, recent approaches leverage deep learning models such as transformers [8] and visionlanguage models [9] to improve detection capabilities.”

    From Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges (2024) · cites this paper as [9]

    Section 3.1.1 says transformers are used for visual-recognition tasks including object detection, and the survey describes approaches that use vision-language models for open-vocabulary object detection. But [9] is attached to the vision-language-model part of a sentence about multi-modal forgeries. The passages support VLM-based object detection, not improved forgery detection; no page numbers are available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • As LLMs expand in the kinds of stimuli they can interpret and the ways they can interact with the world, the space of transferable paradigms increases.

    “As LLMs expand in the kinds of stimuli they can interpret -e.g.visual (OpenAI, 2023b;Zhang, Huang, et al., 2024;Gemini Team et al., 2024) -and the ways in which they can interact with the world -e.g.embodiment and tool use (Mialon et al., 2023) -, the space of transferable paradigms increases.”

    From Machine Psychology (2023) · cites this paper as Zhang, Huang, et al., 2024

  • Prompting LLMs to use code for solving tasks can improve symbolic or numeric reasoning.

    “Regarding improvements in symbolic or numeric reasoning, another technique is to prompt LLMs to use code for solving tasks (Zhang, Ge, et al., 2024).”

    From Machine Psychology (2023) · cites this paper as Zhang, Ge, et al., 2024

    The survey’s Section 9 discusses using LLMs to generate text prompts for vision-language tasks, not prompting LLMs to use code. Searches for symbolic or numeric reasoning and code-based task solving found no passage addressing the attributed technique. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

10 of the 202 references couldn’t be matched to a known paper.

Research works printed but not linked: 17

  1. R. Girshick. Fast r-cnn. In ICCV, pp. 1440-1448, 2015.
  2. T. Chen et al. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607. PMLR, 2020.
  3. A. Radford et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  4. K. He et al. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021. · arXiv
  5. V. Ordonez et al. Im2text: Describing images using 1 million captioned photographs. NeurIPS, 24, 2011.
  6. B. Thomee et al. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
  7. J. Gu et al. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework. arXiv preprint arXiv:2202.06767, 2022. · arXiv
  8. K. Ranasinghe et al. Perceptual grouping in vision-language models. arXiv preprint arXiv:2210.09996, 2022. · arXiv
  9. A. Bulat and G. Tzimiropoulos. Language-aware soft prompt- ing for vision & language foundation models. arXiv preprint arXiv:2210.01115, 2022. · arXiv
  10. M. M. Derakhshani et al. Variational prompt tuning im- proves generalization of vision-language models. arXiv preprint arXiv:2210.02390, 2022. · arXiv
  11. G. Chen et al. Prompt learning with optimal transport for vision- language models. arXiv preprint arXiv:2210.01253, 2022. · arXiv
  12. T.-Y. Wu et al. Protect: Prompt tuning for hierarchical consistency. arXiv preprint arXiv:2306.02240, 2023. · arXiv
  13. Y. Xing et al. Class-aware visual prompt tuning for vision- language pre-trained model. arXiv preprint arXiv:2208.08340, 2022. · arXiv
  14. J. Li et al. Masked unsupervised self-training for zero-shot image classification. arXiv preprint arXiv:2206.02967, 2022. · arXiv
  15. C. Ma et al. Open-vocabulary semantic segmentation with frozen vision-language models. arXiv preprint arXiv:2210.15138, 2022. · arXiv
  16. X. Zhai et al. Scaling vision transformers. In CVPR, pp. 12104- 12113, 2022.
  17. C. Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.

Cited by

Sorted by citations

Related

Authors
Zhang, J., Huang, J., Jin, S., & Lu, S.
Affiliations
Nanyang Technological University (SG)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2024-08
Volume
46
Issue
8
Pages
5625-5644
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2024.3369699
OpenAlex
W4392172801
Cited by
1,051
References
202
Citations by year
2026: 432 · 2025: 486 · 2024: 124 · 2023: 9
FWCI
132.68
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DR-NTU (Nanyang Technological University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
vision-language models, visual recognition, zero-shot learning, transfer learning, knowledge distillation, image-text dataset, pre-training objectives, network architecture, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review