CoChat
Sign Up
PreprintCited by 1

Exploring Vision-Language Models for Imbalanced Learning

Wang, Yidong, Yu, Zhuohao, Wang, Jindong, Heng, Qiang, Chen, Hao, Ye, Wei, Xie, Rui, Xie, Xing +1 more·2023·arXiv
DOI 10.48550/arxiv.2304.01457arXiv 2304.01457
Open this paper in CoChat — free account

Abstract

Vision-Language models (VLMs) that use contrastive language-image pre-training have shown promising zero-shot classification performance. However, their performance on imbalanced dataset is relatively poor, where the distribution of classes in the training dataset is skewed, leading to poor performance in predicting minority classes. For instance, CLIP achieved only 5% accuracy on the iNaturalist18 dataset. We propose to add a lightweight decoder to VLMs to avoid OOM (out of memory) problem caused by large number of classes and capture nuanced features for tail classes. Then, we explore improvements of VLMs using prompt tuning, fine-tuning, and incorporating imbalanced algorithms such as Focal Loss, Balanced SoftMax and Distribution Alignment. Experiments demonstrate that the performance of VLMs can be further boosted when used with decoder and imbalanced methods. Specifically, our improved VLMs significantly outperforms zero-shot classification by an average accuracy of 6.58%, 69.82%, and 6.17%, on ImageNet-LT, iNaturalist18, and Places-LT, respectively. We further analyze the influence of pre-training data size, backbones, and training cost. Our study highlights the significance of imbalanced learning algorithms in face of VLMs pre-trained by huge data. We release our code at https://github.com/Imbalance-VLM/Imbalance-VLM.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.01457

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Wang et al. explored the performance of large vision-language models on imbalanced (long-tailed) tasks, demonstrating the limitation of current large models.

    “Wang et al. [215] first explored the performance of large vision-language models on imbalanced (long-tailed) tasks, which demonstrates the limitation of current large models.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [215]

  • Wang et al. explored different methodologies to enhance performance on imbalanced (long-tailed) tasks.

    “Then, they explored different methodologies to enhance the performance on these tasks.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [215]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 45

  1. Byrd J, Lipton Z (2019) What is the effect of importance weighting in deep learning? In: ICML, PMLR, pp 872-881
  2. Cao K, Wei C, Gaidon A, et al (2019a) Learn- ing imbalanced datasets with label-distribution- aware margin loss. In: NeurIPS
  3. Cao K, Wei C, Gaidon A, et al (2019b) Learning imbalanced datasets with label- distribution-aware margin loss. arXiv preprint arXiv:190607413
  4. Chen H, Fan Y, Wang Y, et al (2022) An embarrassingly simple baseline for imbal- anced semi-supervised learning. arXiv preprint arXiv:221111086
  5. Dehghani M, Djolonga J, Mustafa B, et al (2023) Scaling vision transformers to 22 billion param- eters. arXiv preprint arXiv:230205442
  6. Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations
  7. He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: CVPR, pp 770-778
  8. He K, Chen X, Xie S, et al (2022) Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 16000-16009
  9. Hong Y, Han S, Choi K, et al (2021) Disentan- gling label distribution for long-tailed visual recognition. In: CVPR, pp 6626-6636
  10. Jamal MA, Brown M, Yang MH, et al (2020) Rethinking class-balanced methods for long- tailed visual recognition from a domain adapta- tion perspective. In: CVPR, pp 7610-7619
  11. Kang B, Xie S, Rohrbach M, et al (2019) Decou- pling representation and classifier for long-tailed recognition. In: ICML
  12. Khan SH, Hayat M, Bennamoun M, et al (2017) Cost-sensitive learning of deep feature represen- tations from imbalanced data. IEEE TNNLS 29(8):3573-3587
  13. Li J, Li D, Xiong C, et al (2022) Blip: Bootstrap- ping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learn- ing, PMLR, pp 12888-12900
  14. Lin TY, Goyal P, Girshick R, et al (2017) Focal loss for dense object detection. In: ICCV, pp 2980-2988
  15. Liu Z, Miao Z, Zhan X, et al (2019) Large-scale long-tailed recognition in an open world. In: CVPR, pp 2537-2546
  16. Liu Z, Hu H, Lin Y, et al (2022) Swin trans- former v2: Scaling up capacity and resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12009-12019
  17. Lüddecke T, Ecker A (2022) Image segmentation using text and image prompts. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7086-7096
  18. Ma T, Geng S, Wang M, et al (2021) A simple long-tailed recognition baseline via vision-language model. arXiv preprint arXiv:211114745
  19. Menon AK, Jayasumana S, Rawat AS, et al (2020) Long-tail learning via logit adjustment. In: ICLR
  20. Platt J, Cristianini N, Shawe-Taylor J (1999) Large margin dags for multiclass classification. NIPS 12
  21. Radford A, Kim JW, Hallacy C, et al (2021) Learning transferable visual models from natu- ral language supervision. In: International con- ference on machine learning, PMLR, pp 8748- 8763
  22. Ren J, Yu C, Sheng S, et al (2020) Balanced meta- softmax for long-tailed visual recognition. arXiv preprint arXiv:200710740
  23. Schuhmann C, Beaumont R, Vencu R, et al (2022) Laion-5b: An open large-scale dataset for training next generation image-text models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track
  24. Tan J, Wang C, Li B, et al (2020) Equalization loss for long-tailed object recognition. In: CVPR, pp 11662-11671
  25. Tang K, Huang J, Zhang H (2020) Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurIPS 33
  26. Tian C, Wang W, Zhu X, et al (2022) Vl-ltr: Learning class-wise visual-linguistic representa- tion for long-tailed visual recognition. In: Com- puter Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV, Springer, pp 73-91
  27. Van Horn G, Mac Aodha O, Song Y, et al (2018) The inaturalist species classification and detection dataset. In: CVPR, pp 8769-8778
  28. Vapnik V (1991) Principles of risk minimiza- tion for learning theory. Advances in neural information processing systems 4
  29. Wang J, Lukasiewicz T, Hu X, et al (2021a) Rsg: A simple but effective module for learning imbalanced datasets. In: CVPR, pp 3784-3793
  30. Wang J, Zhang W, Zang Y, et al (2021b) Seesaw loss for long-tailed instance segmentation. In: CVPR, pp 9695-9704
  31. Wang P, Han K, Wei XS, et al (2021c) Contrastive learning based hybrid networks for long-tailed image classification. In: CVPR, pp 943-952
  32. Wang Y, Zhang B, Hou W, et al (2022) Mar- gin calibration for long-tailed visual recogni- tion. In: Asian Conference on Machine Learning (ACML)
  33. Wang YX, Ramanan D, Hebert M (2017) Learning to model the tail. In: NeurIPS, pp 7032-7042
  34. Wei H, Tao L, Xie R, et al (2022) Open- sampling: Exploring out-of-distribution data for re-balancing long-tailed datasets. In: Interna- tional Conference on Machine Learning, PMLR, pp 23615-23630
  35. Xu Z, Yang S, Wang X, et al (2023) Rethink long- tailed recognition with vision transforms. In: ICASSP 2023-2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 1-5
  36. Yang CY, Yang JS, Wang JJ (2009) Margin calibration in svm class-imbalanced learning. Neurocomputing 73(1-3):397-411
  37. Yang L, Jiang H, Song Q, et al (2022) A survey on long-tailed visual recognition. IJCV pp 1-36
  38. Yang Y, Xu Z (2020) Rethinking the value of labels for improving class-imbalanced learning. In: NeurIPS
  39. Yin X, Yu X, Sohn K, et al (2019) Feature trans- fer learning for face recognition with under- represented data. In: CVPR
  40. Yu J, Wang Z, Vasudevan V, et al (2022) Coca: Contrastive captioners are image-text founda- tion models. arXiv preprint arXiv:220501917
  41. Zhang S, Li Z, Yan S, et al (2021) Distribution alignment: A unified framework for long-tail visual recognition. In: CVPR
  42. Zhou B, Lapedriza A, Khosla A, et al (2017) Places: A 10 million image database for scene recognition. IEEE TPAMI 40(6):1452-1464
  43. Zhou B, Cui Q, Wei XS, et al (2020) Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In: CVPR
  44. Zhou K, Yang J, Loy CC, et al (2022a) Con- ditional prompt learning for vision-language models. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recog- nition, pp 16816-16825
  45. Zhou K, Yang J, Loy CC, et al (2022b) Learning to prompt for vision-language models. Interna- tional Journal of Computer Vision 130(9):2337- 2348

Cited by

Sorted by citations

Authors
Wang, Y., Yu, Z., Wang, J., Heng, Q., Chen, H., Ye, W., Xie, R., Xie, X., & Zhang, S.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2304.01457
arXiv
2304.01457
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review