CoChat
Sign Up
Cited by 40Domain Adaptation and Few-Shot LearningMultimodal Machine Learning ApplicationsAdvanced Neural Network Applications

Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

Open access
D’Alessandro, Marco, Alonso, Alberto, Calabrés, Enrique, Galar, Mikel·2023·2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
DOI 10.1109/iccvw60793.2023.00364
Open this paper in CoChat — free accountOpen PDF

Abstract

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a predefined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccvw60793.2023.00364

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Touqeer Ahmad, Akshay Raj Dhamija, Steve Cruz, Ryan Ra- binowitz, Chunchun Li, Mohsen Jafarzadeh, and Terrance E Boult. Few-shot class incremental learning leveraging self- supervised features. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3900-3910, 2022.
  2. Ali Ayub and Alan R Wagner. Tell me what this is: Few-shot incremental object learning by a robot. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8344-8350. IEEE, 2020.
  3. Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF Inter- national conference on computer vision, pages 9516-9525, 2021.
  4. Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classifi- cation. arXiv preprint arXiv:1904.04232, 2019. · arXiv
  5. Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Ku- mar Roy, Lars Petersson, and Mehrtash Harandi. Semantic- aware knowledge distillation for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2534-2543, 2021.
  6. Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe, Pengfei Fang, Christian Simon, Lars Petersson, and Mehrtash Harandi. Synthesized feature based few-shot class- incremental learning on a mixture of subspaces. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 8661-8670, 2021.
  7. Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 386-402. Springer, 2020.
  8. Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few- shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12365-12375, 2020.
  9. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126- 1135. PMLR, 2017.
  10. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021. · arXiv
  11. Donald Olding Hebb. The organization of behavior: A neu- ropsychological theory. Psychology Press, 2005.
  12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
  13. Neil Houlsby, Andreea Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. K-adapter: Infusing knowledge into pre-trained models with adapters. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 56-61, 2020.
  14. Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning, July 2022. arXiv:2203.12119 [cs]. · arXiv
  15. Yifan Jiang, Han Chen, Hanseok Ko, and David K Han. Few-shot learning for ct scan based covid-19 diagnosis. In ICASSP 2021-2021 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 1045- 1049. IEEE, 2021.
  16. Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi- modal prompt learning. arXiv preprint arXiv:2210.03117, 2022. · arXiv
  17. Zhizhong Li and Derek Hoiem. Learning without forget- ting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
  18. Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Ming- sheng Long, and Han Hu. Negative margin matters: Un- derstanding margin in few-shot classification. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 438-455. Springer, 2020.
  19. Binghao Liu, Boyu Yang, Lingxi Xie, Ren Wang, Qi Tian, and Qixiang Ye. Learnable distribution calibration for few-shot class-incremental learning. arXiv preprint arXiv:2210.00232, 2022. · arXiv
  20. Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, An- tonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Be- louadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Workshop, 2021.
  21. Arun Mallya and Svetlana Lazebnik. Packnet: Adding multi- ple tasks to a single network by iterative pruning. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
  22. Pratik Mazumder, Pravendra Singh, and Piyush Rai. Few-shot lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2337-2345, 2021.
  23. Michael McCloskey and Neal J Cohen. Catastrophic interfer- ence in connectionist networks: The sequential learning prob- lem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
  24. Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Con- tinual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-16144, 2021.
  25. Marcin Przewięźlikowski, Przemysław Przybysz, Jacek Tabor, M Zięba, and Przemysław Spurek. Hypermaml: Few-shot adaptation of deep models with hypernetworks. arXiv preprint arXiv:2205.15745, 2022. · arXiv
  26. Haoxuan Qu, Hossein Rahmani, Li Xu, Bryan Williams, and Jun Liu. Recent advances of continual learning in computer vision: An overview. arXiv preprint arXiv:2109.11369, 2021. · arXiv
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
  28. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
  29. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Raz- van Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. · arXiv
  30. James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Contin- ual decomposed attention-based prompting for rehearsal-free continual learning. arXiv preprint arXiv:2211.13218, 2022. · arXiv
  31. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural informa- tion processing systems, 30, 2017.
  32. Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
  33. Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12183- 12192, 2020.
  34. Vishal Thengane, Salman Khan, Munawar Hayat, and Fahad Khan. Clip model is an efficient continual learner. arXiv preprint arXiv:2210.03114, 2022. · arXiv
  35. Gido M van de Ven, Tinne Tuytelaars, and Andreas S To- lias. Three types of incremental learning. Nature Machine Intelligence, pages 1-13, 2022.
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  37. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
  38. Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
  39. Zifeng Wang, Tong Jian, Kaushik Chowdhury, Yanzhi Wang, Jennifer Dy, and Stratis Ioannidis. Learn-prune-share for lifelong learning. In 2020 IEEE International Conference on Data Mining (ICDM), pages 641-650. IEEE, 2020.
  40. Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Is- rael, October 23-27, 2022, Proceedings, Part XXVI, pages 631-648. Springer, 2022.
  41. Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jen- nifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
  42. Xinlei Xu, Zhe Wang, Zhiling Fu, Wei Guo, Ziqiu Chi, and Dongdong Li. Flexible few-shot class-incremental learning with prototype container. Neural Computing and Applications, pages 1-15, 2023.
  43. Li Yin, Juan M Perez-Rua, and Kevin J Liang. Sylph: A hypernetwork framework for incremental few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9035-9045, 2022.
  44. Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable net- works. arXiv preprint arXiv:1708.01547, 2017. · arXiv
  45. Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. arXiv preprint arXiv:1802.01557, 2018. · arXiv
  46. Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995. PMLR, 2017.
  47. Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with contin- ually evolved classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12455-12464, 2021.
  48. Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shil- iang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9046-9056, 2022.
  49. Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan. Few-shot class-incremental learning by sampling multi-phase tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  50. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
  51. Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. Self-promoted prototype refinement for few-shot class- incremental learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6801-6810, 2021.
  52. Yixiong Zou, Shanghang Zhang, Yuhua Li, and Ruixuan Li. Margin-based few-shot class-incremental learning with class- level overfitting mitigation. arXiv preprint arXiv:2210.04524, 2022. · arXiv

Cited by

Sorted by citations

Authors
D’Alessandro, M., Alonso, A., Calabrés, E., & Galar, M.
Affiliations
  • Neuraptic AI
  • Universidad Pública de Navarra (UPNA) (ES)
ORCID
Venue
2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
Publisher
IEEE
Published
2023-10-02
Pages
3385-3395
ISSN
2473-9936 (ISSN-L)
DOI
10.1109/iccvw60793.2023.00364
OpenAlex
W4390189747
Cited by
40
References
11
Citations by year
2026: 12 · 2025: 19 · 2024: 9
FWCI
7.09
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Computer Vision Workshops (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Academica-e (Universidad Pública de Navarra) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
few-shot class-incremental learning, continual learning, parameter-efficient learning, CLIP, multimodal learning, learnable prompts, catastrophic forgetting, vision-language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review