Maintaining Discrimination and Fairness in Class Incremental Learning
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Deep neural networks (DNNs) have been applied in class incremental learning, which aims to solve common real-world problems of learning new classes continually. One drawback of standard DNNs is that they are prone to catastrophic forgetting. Knowledge distillation (KD) is a commonly used technique to alleviate this problem. In this paper, we demonstrate it can indeed help the model to output more discriminative results within old classes. However, it cannot alleviate the problem that the model tends to classify objects into new classes, causing the positive effect of KD to be hidden and limited. We observed that an important factor causing catastrophic forgetting is that the weights in the last fully connected (FC) layer are highly biased in class incremental learning. In this paper, we propose a simple and effective solution motivated by the aforementioned observations to address catastrophic forgetting. Firstly, we utilize KD to maintain the discrimination within old classes. Then, to further maintain the fairness between old classes and new classes, we propose Weight Aligning (WA) that corrects the biased weights in the FC layer after normal training process. Unlike previous work, WA does not require any extra parameters or a validation set in advance, as it utilizes the information provided by the biased weights themselves. The proposed method is evaluated on ImageNet-1000, ImageNet-100, and CIFAR-100 under various settings. Experimental results show that the proposed method can effectively alleviate catastrophic forgetting and significantly outperform state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“We evaluate the methods on ImageNet ILSVRC 2012 [27] and CIFAR-100 [19], which are widely used in the study of class incremental learning [5, 26, 32].”
MethodLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“Knowledge distillation (KD) [12] is another strategy, which has also been widely used in this field [5, 20, 39].”
Method(2017) · Cited by 11,625
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 2×
“per class, and 10,000 images for evaluating with 100 images per class. Our method are implemented with Pytorch [24]. The code will be made publicly available. For ImageNet, we adopt a 18-layer ResNet [10,11]. We use SGD to train our model and set the batch size to 256. The learning rate starts from 0.1 and reduces to…”
BackgroundMethodInfluential
Cited by
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
“Label Space LwF [252], iCaRL [356], GEM [277], A-GEM [65], EEIL [53], BiC [457], WA [500], DER++ [46], ScaIL [33], S&B [218], SS-IL [11], Coil [506]”
BackgroundInfluentialDER: Dynamically Expandable Representation for Class Incremental Learning
Yan, S., Xie, J., & He, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00303 · Cited by 660
GDumb: A Simple Approach that Questions Our Progress in Continual Learning
Prabhu, A., Torr, P. H. S., & Dokania, P. K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_31 · Cited by 465
Prototype Augmentation and Self-Supervision for Incremental Learning
Zhu, F., Zhang, X.-Y., Wang, C., Yin, F., & Liu, C.-L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00581 · Cited by 427
“More recently, the imbalance problem between the previous and current tasks has been found to be constituting a key challenge for CIL, and several works, such as EEIL [6], BiC [50], UCIR [22] and WA [56] were proposed to reduce the bias towards currents tasks.”
Online continual learning in image classification: An empirical survey
Mai, Z., Li, R., Jeong, J., Quispe, D., Kim, H., & Sanner, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.10.021 · Cited by 410
Class-Incremental Learning: A Survey
Zhou, D.-W., Wang, Q.-W., Qi, Z.-H., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429383 · Cited by 376
“WA [112] further normalizes the weight after every optimization step.”
BackgroundMethodInfluentialDyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
Douillard, A., Rame, A., Couairon, G., & Cord, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00907 · Cited by 357
FOSTER: Feature Boosting and Compression for Class-Incremental Learning
Wang, F.-Y., Zhou, D.-W., Ye, H.-J., & Zhan, D.-C. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19806-9_23 · Cited by 327
Forward Compatible Few-Shot Class-Incremental Learning
Zhou, D.-W., Wang, F.-Y., Ye, H.-J., Ma, L., Pu, S., & Zhan, D.-C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00884 · Cited by 281
PLOP: Learning without Forgetting for Continual Semantic Segmentation
Douillard, A., Chen, Y., Dapogny, A., & Cord, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00403 · Cited by 267
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning
Cheraghian, A., Rahman, S., Fang, P., Roy, S. K., Petersson, L., & Harandi, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00256 · Cited by 201
“[41] employed knowledge distillation, at first, to keep the discrimination of old classes.”
MethodSS-IL: Separated Softmax for Incremental Learning
Ahn, H., Kwak, J., Lim, S., Bang, H., Kim, H., & Moon, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00088 · Cited by 177
“BiC [31] corrects the bias of scores by additionally training a bias correction layer, and WA [35] corrects the biased weights in the FC layer based on the norm of each weight.”
BackgroundMethodInfluentialConstrained Few-shot Class-incremental Learning
Hersche, M., Karunaratne, G., Cherubini, G., Benini, L., Sebastian, A., & Rahimi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00885 · Cited by 167
“In a similar vein, a weight aligning method has been proposed to align the norms of the weight vectors for novel classes to those for past classes [57].”
MethodFew-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks
Zhou, D.-W., Ye, H.-J., Ma, L., Xie, D., Pu, S., & Zhan, D.-C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3200865 · Cited by 162
“Other works consider saving embeddings instead of raw images [38], using generative models for data rehearsal [39], task-wise adaptation [40], and output normalization [18], [41], [42].”
BackgroundAn Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120
“Zhao et al. [79] combined weight adjustment and knowledge distillation in order to balance the new and old knowledge.”
BackgroundMgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning
Zhao, H., Fu, Y., Kang, M., Tian, Q., Wu, F., & Li, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3133897 · Cited by 115
Replay in Deep Learning: Current Approaches and Missing Biological Elements
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, H. T., Sejnowski, T. J., & Kanan, C. (2021) · Neural Computation · doi:10.1162/neco_a_01433 · Cited by 106
“…Chaudhry et al., 2019; Riemer et al., 2019; Aljundi, Belilovsky et al., 2019; Aljundi, Lin, Goujaud, & Bengio, 2019; He, Mao, Shao, & Zhu, 2020; Zhao, Xiao, Gan, Zhang, & Xia, 2020; Kurle, Cseke, Klushyn, van der Smagt, & Günnemann, 2020; Chrysakis & Moens, 2020; Kim, Jeong, & Kim, 2020; Tao,…”
(2021) · International Conference on Learning Representations · Cited by 65
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning
Kalla, J., & Biswas, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19806-9_25 · Cited by 45
“Several state-of-art-works [38,45,2,12,3] proposed different techniques to address the class imbalance problem in CIL like rescaling scores or balanced finetuning of classifiers, etc.”
MethodFew-Shot Class Incremental Learning Leveraging Self-Supervised Features
Ahmad, T., Dhamija, A. R., Cruz, S., Rabinowitz, R., Li, C., Jafarzadeh, M., & Boult, T. E. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00435 · Cited by 41
Continual Learning for Generative Retrieval over Dynamic Corpora
Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., & Cheng, X. (2023) · Proceedings of the 32nd ACM International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614821 · Cited by 40
“Existing CL approaches [13] can be categorized into: (i) replay methods, maintaining a subset of previous samples and training models together with samples in the new session [18, 52]; (ii) regularization-based methods, regularizing the model parameters to enable important parameters concerning the previous tasks to b…”
MethodA Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Belouadah, E., Popescu, A., & Kanellos, I. (2020) · arXiv · doi:10.48550/arxiv.2011.01844 · Cited by 235
Metadata
- Authors
- Zhao, B., Xiao, X., Gan, G., Zhang, B., & Xia, S.-T.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 13205-13214
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3034856281
- Cited by
- 494
- References
- 43
- FWCI
- 22.27
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.