Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning
Open accessAbstract
Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation process. Furthermore, we propose a method based on an attention mechanism on multiple parallel embeddings of visual data to align visual and semantic vectors, which reduces issues related to catastrophic forgetting. Via experiments on MiniImageNet, CUB200, and CIFAR100 dataset, we establish new state-of-the-art results by outperforming existing approaches.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cheraghian et al. [17] proposed a semantic-aware KD framework.
“Cheraghian et al. [17] proposed a semantic-aware KD framework.”
From Few-Shot Class-Incremental Learning for Classification and Object Detection: A Survey (2025) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_33 · Cited by 1,091 · see citing works
Incremental Learning of Object Detectors without Catastrophic Forgetting
Shmelkov, K., Schmid, C., & Alahari, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.368 · Cited by 559 · see citing works
Low-Shot Learning with Imprinted Weights
Qi, H., Brown, M., & Lowe, D. G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00610 · Cited by 539 · see citing works
Flagged first, then by citations
Research works printed but not linked: 31
- Francisco M. Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-End Incre- mental Learning. In Vittorio Ferrari, Martial Hebert, Cris- tian Sminchisescu, and Yair Weiss, editors, ECCV 2018 -Eu- ropean Conference on Computer Vision, Munich, Germany, Sept. 2018.
- Francisco M. Castro, Manuel J. Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Alahari. End-to-End Incremental Learning. In Proceedings of the European Con- ference on Computer Vision (ECCV), September 2018.
- Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero- shot learning for object recognition in the wild. In Bas- tian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision -ECCV 2016, pages 52-68, Cham, 2016. Springer International Publishing.
- Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Mitigating the hubness problem for zero-shot learning of 3d objects. In British Machine Vision Conference (BMVC'19), 2019.
- Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learning for 3d point cloud classification. In 2020 IEEE Winter Conference on Applica- tions of Computer Vision (WACV), pages 912-922, 2020.
- Pengfei Fang, Jieming Zhou, Soumava Kumar Roy, Lars Pe- tersson, and Mehrtash Harandi. Bilinear Attention Networks for Person Retrieval. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), October 2019.
- S. Gidaris and N. Komodakis. Dynamic Few-Shot Visual Learning Without Forgetting. In 2018 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 4367-4375, 2018.
- Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
- Jie Hong, Pengfei Fang, Weihao Li, Zhang Tong, Christian Simon, Lars Petersson, and Mehrtash Harandi. Reinforced Attention for Few-Shot Learning and Beyond. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
- Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a Unified Classifier Incrementally via Rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
- Z. Li and D. Hoiem. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947, 2018.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111-3119.
- Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A Simple Neural Attentive Meta-Learner. In Inter- national Conference on Learning Representations, 2018.
- Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees G. M. Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In European Conference on Computer Vision, August 2020.
- Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational Continual Learning. In In- ternational Conference on Learning Representations, 2018.
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global Vectors for Word Representa- tion. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.
- Shafin Rahman, Salman Khan, and Nick Barnes. Transduc- tive learning for zero-shot object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), October 2019.
- Shafin Rahman, Salman Khan, Nick Barnes, and Fa- had Shahbaz Khan. Any-shot object detection. In Computer Vision -ACCV 2020, pages 89-106, Cham, 2021. Springer International Publishing.
- Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S. Zemel. Incremental Few-Shot Learning with Attention At- tractor Networks, booktitle= Advances in Neural Informa- tion Processing Systems (NeurIPS), year = 2019,.
- Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, , and Gerald Tesauro. Learning to Learn Without Forgetting By Maximizing Transfer and Min- imizing Interference. In International Conference on Learn- ing Representations, 2019.
- Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In Ad- vances in Neural Information Processing Systems 30, pages 2990-2999, 2017.
- Christian Simon, Piotr Koniusz, and Mehrtash Harandi. On Learning the Geodesic Path for Incremental Learning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2021.
- Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-Shot Class- Incremental Learning. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2020.
- Gido M. van de Ven and Andreas S. Tolias. Three Scenarios for Continual Learning. CoRR, abs/1904.07734, 2019. · arXiv
- Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching Networks for One Shot Learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neu- ral Information Processing Systems 29, pages 3630-3638.
- Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large Scale Incre- mental Learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2019.
- Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embed- dings for zero-shot classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), June 2016.
- Sung Whan Yoon, Do-Yeon Kim, Jun Seo, and Jaekyun Moon. Xtarnet: Learning to Extract Task-Adaptive Rep- resentation for Incremental Few-Shot Learning. In In Pro- ceedings of the 37th International Conference on Machine Learning (ICML), 2020.
- Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual Learning Through Synaptic Intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995, 06-11 Aug 2017.
- J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C. . Jay Kuo. Class-incremental Learning via Deep Model Consolidation. In 2020 IEEE Winter Con- ference on Applications of Computer Vision (WACV), pages 1120-1129, 2020.
- B. Zhao, X. Xiao, G. Gan, B. Zhang, and S. T. Xia. Maintain- ing Discrimination and Fairness in Class Incremental Learn- ing. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13205-13214, 2020.
Cited by
Sorted by citations
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113 · see citing works
“Besides, auxiliary information such as semantic word vectors [262], [263] and sketch exemplars [264] can be incorporated to enrich the limited training samples.”
BackgroundFew-Shot Class-Incremental Learning for Classification and Object Detection: A Survey
Zhang, J., Liu, L., Silvén, O., Pietikäinen, M., & Hu, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3529038 · Cited by 84 · see citing works
“The role of attention modules used in many FSCIL approaches [14], [16], [17], [73], [74] is diverse.”
InfluentialCatastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54 · see citing works
Human-inspired Perspectives: A Survey on AI Long-term Memory
He, Z., Lin, W., Zheng, H., Zhang, F., Jones, M. W., Aitchison, L., Xu, X., Liu, M., Kristensson, P. O., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.00489 · Cited by 3 · see citing works
“For example, both Cheraghian et al. [127] and Ni et al. [130] used fixed semantic embeddings to represent class information, which ensured a degree of separability between data from different classes.”
- Authors
- Cheraghian, A., Rahman, S., Fang, P., Roy, S. K., Petersson, L., & Harandi, M.
- Affiliations
- Australian National University (AU)
- North South University (BD)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 2534-2543
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3134911507
- Cited by
- 201
- References
- 7
- Citations by year
- 2026: 11 · 2025: 44 · 2024: 55 · 2023: 53 · 2022: 30 · 2021: 8
- FWCI
- 16.86
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- few-shot class-incremental learning, knowledge distillation, semantic information, word embeddings, attention mechanism, catastrophic forgetting, visual-semantic alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
