CoChat
Sign Up
Cited by 39Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Data Classification

Uncertainty-Aware Distillation for Semi-Supervised Few-Shot Class-Incremental Learning

Open access
Cui, Yawen, Deng, Wanxia, Chen, Haoyu, Liu, Li·2024·IEEE Transactions on Neural Networks and Learning Systems
DOI 10.1109/tnnls.2023.3277018
Open this paper in CoChat — free accountOpen PDF

Abstract

Given a model well-trained with a large-scale base dataset, few-shot class-incremental learning (FSCIL) aims at incrementally learning novel classes from a few labeled samples by avoiding overfitting, without catastrophically forgetting all encountered classes previously. Currently, semi-supervised learning technique that harnesses freely available unlabeled data to compensate for limited labeled data can boost the performance in numerous vision tasks, which heuristically can be applied to tackle issues in FSCIL, i.e., the semi-supervised FSCIL (Semi-FSCIL). So far, very limited work focuses on the Semi-FSCIL task, leaving the adaptability issue of semi-supervised learning to the FSCIL task unresolved. In this article, we focus on this adaptability issue and present a simple yet efficient Semi-FSCIL framework named uncertainty-aware distillation with class-equilibrium (UaD-ClE), encompassing two modules: uncertainty-aware distillation (UaD) and class equilibrium (ClE). Specifically, when incorporating unlabeled data into each incremental session, we introduce the ClE module that employs a class-balanced self-training (CB_ST) to avoid the gradual dominance of easy-to-classified classes on pseudo-label generation. To distill reliable knowledge from the reference model, we further implement the UaD module that combines uncertainty-guided knowledge refinement with adaptive distillation. Comprehensive experiments on three benchmark datasets demonstrate that our method can boost the adaptability of unlabeled data with the semi-supervised learning technique in FSCIL tasks. The code is available at https://github.com/yawencui/UaD-ClE.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tnnls.2023.3277018

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in NeurIPS, 2012, pp. 1097- 1105. 1
  2. K. Tang, Y. Ma, D. Miao, P. Song, Z. Gu, Z. Tian, and W. Wang, "Decision fusion networks for image classification," IEEE Transactions on Neural Networks and Learning Systems, 2022. 1
  3. S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," in NeurIPS, 2015, pp. 91-99. 1
  4. W. Zhou, Q. Guo, J. Lei, L. Yu, and J.-N. Hwang, "Irfr-net: interactive recursive feature-reshaping network for detecting salient objects in rgb-d images," IEEE Transactions on Neural Networks and Learning Systems, 2021. 1
  5. J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, "Dual attention network for scene segmentation," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3146- 3154. 1
  6. J. Fu, J. Liu, J. Jiang, Y. Li, Y. Bao, and H. Lu, "Scene segmentation with dual relation-aware attention network," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 6, pp. 2547-2560, 2020.
  7. K. Zhu, Y. Cao, W. Zhai, J. Cheng, and Z.-J. Zha, "Self-promoted prototype refinement for few-shot class-incremental learning," in CVPR, 2021, pp. 6801-6810. 1, 2, 3, 4, 7, 8, 9, 10
  8. X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, and Y. Gong, "Few-shot class-incremental learning," in CVPR, 2020, pp. 12 183-12 192. 1, 2, 3, 6, 7, 9, 10, 12
  9. C. Zhang, N. Song, G. Lin, Y. Zheng, P. Pan, and Y. Xu, "Few-shot incremental learning with continually evolved classifiers," in CVPR, 2021, pp. 12 455-12 464. 1, 2, 3, 4, 6, 7, 9, 10
  10. K. Chen and C.-G. Lee, "Incremental few-shot learning via vector quantization in deep embedded space," in ICLR, 2021. 1, 2, 9
  11. D.-W. Zhou, F.-Y. Wang, H.-J. Ye, L. Ma, S. Pu, and D.-C. Zhan, "Forward compatible few-shot class-incremental learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9046-9056. 1, 2, 9
  12. Z. Chi, L. Gu, H. Liu, Y. Wang, Y. Yu, and J. Tang, "Metafscil: A meta-learning approach for few-shot class incremental learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 166-14 175. 1, 2, 9
  13. H. Liu, L. Gu, Z. Chi, Y. Wang, Y. Yu, J. Chen, and J. Tang, "Few-shot class-incremental learning via entropy-regularized data-free replay," in European Conference on Computer Vision. Springer, 2022, pp. 146- 162. 1, 2, 9
  14. T. Chen, S. Wu, X. Yang, Y. Xu, and H.-S. Wong, "Semantic regularized class-conditional gans for semi-supervised fine-grained image synthesis," IEEE Transactions on Multimedia, 2021. 1, 2
  15. G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, "Continual lifelong learning with neural networks: A review," Neural Networks, vol. 113, pp. 54-71, 2019. 1
  16. J. E. Van Engelen and H. H. Hoos, "A survey on semi-supervised learning," Machine Learning, vol. 109, no. 2, pp. 373-440, 2020. 1, 4
  17. Y. Cui, W. Xiong, M. Tavakolian, and L. Liu, "Semi-supervised few- shot class-incremental learning," in ICIP. IEEE, 2021, pp. 1239-1243. 1, 3, 4, 5, 6, 7, 9, 10
  18. Y. Cui, W. Deng, X. Xu, Z. Liu, Z. Liu, M. Pietikäinen, and L. Liu, "Uncertainty-guided semi-supervised few-shot class-incremental learning with knowledge distillation," IEEE Transactions on Multimedia, 2022. 1, 2, 3, 7, 9
  19. B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, "Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling," NeurIPS, vol. 34, 2021. 2, 3, 5
  20. N. Lai, M. Kan, C. Han, X. Song, and S. Shan, "Learning to learn adaptive classifier-predictor for few-shot learning," IEEE transactions on neural networks and learning systems, vol. 32, no. 8, pp. 3458-3470, 2020. 2
  21. H.-G. Jung and S.-W. Lee, "Few-shot learning with geometric constraints," IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 11, pp. 4660-4672, 2020. 2
  22. X. Yu and Y. Aloimonos, "Attribute-based transfer learning for object categorization with zero/one training example," in ECCV, 2010, pp. 127- 140. 2
  23. O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra et al., "Matching networks for one shot learning," in NeurPIS, 2016, pp. 3630-3638. 2, 6
  24. J. Snell, K. Swersky, and R. Zemel, "Prototypical networks for few-shot learning," in NeurIPS, 2017, pp. 4080-4090. 2
  25. C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in ICML, 2017, pp. 1126-1135. 2
  26. M. A. Jamal and G.-J. Qi, "Task agnostic meta-learning for few-shot learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 719-11 727. 2
  27. A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, "Meta-learning with memory-augmented neural networks," in ICML, 2016, pp. 1842-1850. 2
  28. S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, "Learning a unified classifier incrementally via rebalancing," in CVPR, 2019, pp. 831-839. 2, 5, 6, 7, 9, 12
  29. H. Zhao, H. Wang, Y. Fu, F. Wu, and X. Li, "Memory efficient class- incremental learning for image classification," IEEE Transactions on Neural Networks and Learning Systems, 2021. 2
  30. D.-W. Zhou, Y. Yang, and D.-C. Zhan, "Learning to classify with incremental new class," IEEE Transactions on Neural Networks and Learning Systems, 2021. 2
  31. Y. Liu, X. Hong, X. Tao, S. Dong, J. Shi, and Y. Gong, "Model behavior preserving for class-incremental learning," IEEE Transactions on Neural Networks and Learning Systems, 2022. 2
  32. A. Cheraghian, S. Rahman, P. Fang, S. K. Roy, L. Petersson, and M. Harandi, "Semantic-aware knowledge distillation for few-shot class- incremental learning," in CVPR, 2021, pp. 2534-2543. 3, 9
  33. Z. Song, X. Yang, Z. Xu, and I. King, "Graph-based semi-supervised learning: A comprehensive review," IEEE Transactions on Neural Networks and Learning Systems, 2022. 3
  34. D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, "Mixmatch: A holistic approach to semi-supervised learning," in NeurIPS, 2019. 3, 12
  35. D.-H. Lee et al., "Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks," in ICML Workshop, vol. 3, no. 2, 2013. 3
  36. K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying semi- supervised learning with consistency and confidence," NeurIPS, vol. 33, pp. 596-608, 2020. 3, 12
  37. A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, "Label propagation for deep semi-supervised learning," in CVPR, 2019, pp. 5070-5079. 3
  38. M. N. Rizve, K. Duarte, Y. S. Rawat, and M. Shah, "In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection frame- work for semi-supervised learning," arXiv preprint arXiv:2101.06329, 2021. 3 · arXiv
  39. X. Zhu, Z. Ghahramani, and J. D. Lafferty, "Semi-supervised learning using gaussian fields and harmonic functions," in ICML, 2003, pp. 912- 919. 3
  40. B. Liu, Z. Wu, H. Hu, and S. Lin, "Deep metric transfer for label propagation with limited annotated data," in ICCV Workshop, 2019, pp. 0-0. 3
  41. A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" Advances in neural information processing systems, vol. 30, 2017. 3
  42. C. Louizos and M. Welling, "Structured and efficient variational deep learning with matrix gaussian posteriors," in ICML, 2016, pp. 1708- 1716. 3
  43. G. Wang, W. Li, S. Ourselin, and T. Vercauteren, "Automatic brain tumor segmentation using convolutional neural networks with test- time augmentation," in International MICCAI Brainlesion Workshop. Springer, 2018, pp. 61-72. 3
  44. J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher et al., "A survey of uncertainty in deep neural networks," arXiv preprint arXiv:2107.03342, 2021. 3, 5 · arXiv
  45. Y. Grandvalet and Y. Bengio, "Semi-supervised learning by entropy minimization," in NeurIPS, 2004, pp. 529-536. 4, 9
  46. A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, "Efficient lifelong learning with a-gem," in ICLR, 2018. 6
  47. H. Robbins and S. Monro, "A stochastic approximation method," The annals of mathematical statistics, pp. 400-407, 1951. 6
  48. F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, "End-to-end incremental learning," in ECCV, 2018, pp. 233-248. 9
  49. S. Dong, X. Hong, X. Tao, X. Chang, X. Wei, and Y. Gong, "Few-shot class-incremental learning via relation knowledge distillation," in AAAI, 2021, pp. 1255-1263. 9
  50. A. Cheraghian, S. Rahman, S. Ramasinghe, P. Fang, C. Simon, L. Petersson, and M. Harandi, "Synthesized feature based few-shot class- incremental learning on a mixture of subspaces," in ICCV, 2021, pp. 8661-8670. 9
  51. V. D. M. Laurens and G. Hinton, "Visualizing data using t-sne," The Journal of Machine Learning Research, vol. 9, p. 2579-2605, 2008. 7
  52. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," in NeurIPS Workshop, 2015. 8

Cited by

Sorted by citations

Authors
Cui, Y., Deng, W., Chen, H., & Liu, L.
Affiliations
  • University of Oulu (FI)
  • National University of Defense Technology (CN)
ORCID
Venue
IEEE Transactions on Neural Networks and Learning Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2024-10
Volume
35
Issue
10
Pages
14259-14272
ISSN
  • 2162-237X (ISSN-L)
  • 2162-2388
DOI
10.1109/tnnls.2023.3277018
OpenAlex
W4378804869
Cited by
39
References
5
Citations by year
2026: 6 · 2025: 18 · 2024: 10 · 2023: 5
FWCI
3.45
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Key Research and Development Program of China (2021YFB3100800)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
few-shot class-incremental learning, semi-supervised learning, knowledge distillation, uncertainty estimation, catastrophic forgetting, pseudo-labeling, unlabeled data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review