CoChat
Sign Up
Cited by 1,313Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Data Classification

Large Scale Incremental Learning

Open access
Wu, Yue, Chen, Yinpeng, Wang, Lijuan, Ye, Yuancheng, Liu, Zicheng, Guo, Yandong, Fu, Yun·2019·2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2019.00046
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern machine learning suffers from \textit{catastrophic forgetting} when learning new classes incrementally. The performance dramatically degrades due to the missing data of old classes. Incremental learning methods have been proposed to retain the knowledge acquired from the old classes, by using knowledge distilling and keeping a few exemplars from the old classes. However, these methods struggle to \textbf{scale up to a large number of classes}. We believe this is because of the combination of two factors: (a) the data imbalance between the old and new classes, and (b) the increasing number of visually similar classes. Distinguishing between an increasing number of visually similar classes is particularly challenging, when the training data is unbalanced. We propose a simple and effective method to address this data imbalance issue. We found that the last fully connected layer has a strong bias towards the new classes, and this bias can be corrected by a linear model. With two bias parameters, our method performs remarkably well on two large datasets: ImageNet (1000 classes) and MS-Celeb-1M (10000 classes), outperforming the state-of-the-art algorithms by 11.1\% and 13.2\% respectively.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2019.00046

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The inconsistency of LwF is due to its severe classification bias towards later tasks.

    “The inconsistency of LwF is due to its severe classification bias towards later tasks as discussed in BiC [70].”

    From Open-world continual learning: Unifying novelty detection and continual learning (2025) · cites this paper as [70]

    Section 4, “Diagnosis: FC Layer is Biased” (page unavailable), finds a strong bias toward new classes in the paper’s baseline classifier. Section 6.3, “Comparison on Large Datasets” (page unavailable), reports declining LwF accuracy, but does not establish that LwF’s inconsistency is caused by that bias; unlike the tested baseline, LwF uses no old-class exemplars. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Class IL algorithms provide the most interesting results in recent works from literature.

    “We focus on class IL algorithms which provide the most interesting results in recent works from literature [7, 8, 14, 23, 24, 28, 56, 75].”

    From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [75]

  • The other methods with strong performance were ScaIL (GIL = -3.7), BiC (GIL = -4.03) and LUCIR CNN (GIL = -4.13).

    “The other methods with strong performance were all proposed recently: ScaIL [8] ( GIL = *3 : 7 ), BiC [75] ( GIL = *4 : 03 ) and LUCIR CNN [28] ( GIL = *4 : 13 ).”

    From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [75]

  • BiC can only function if a memory K is available, because a validation set is needed to optimize the bias removal layer.

    “Since a validation set is needed to optimize the bias removal layer, BiC can only function if a memory K is available.”

    From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [75]

  • The model is biased towards new classes, leading it to predict past images as belonging to new classes.

    “This is explained in [7, 8, 75] by the fact that the model is biased towards new classes, leading to predict past images as belonging to new classes.”

    From A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020) · cites this paper as [75]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. Francisco M. Castro, Manuel J. Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Alahari. End-to-end in- cremental learning. In The European Conference on Com- puter Vision (ECCV), September 2018.
  2. Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In Advances in neural information processing systems, pages 409-415, 2001.
  3. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
  4. Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. MS-Celeb-1M: A dataset and benchmark for large scale face recognition. In ECCV, 2016.
  5. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
  6. Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122, 2016. · arXiv
  7. Ilja Kuzborskij, Francesco Orabona, and Barbara Caputo. From n to n+ 1: Multiclass transfer incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3358-3365, 2013.
  8. Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, pages 614- 629. Springer, 2016.
  9. David Lopez-Paz et al. Gradient episodic memory for contin- ual learning. In Advances in Neural Information Processing Systems, pages 6470-6479, 2017.
  10. Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Gen- eralizing to new classes at near-zero cost. IEEE transactions on pattern analysis and machine intelligence, 35(11):2624- 2637, 2013.
  11. Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 31(4):497-508, 2001.
  12. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Raz- van Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. · arXiv
  13. Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pages 2994-3003, 2017.
  14. Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
  15. Gan Sun, Yang Cong, Ji Liu, Lianqing Liu, Xiaowei Xu, and Haibin Yu. Lifelong metric learning. IEEE transactions on cybernetics, (99):1-12, 2018.
  16. Gan Sun, Yang Cong, and Xiaowei Xu. Active lifelong learn- ing with" watchdog". In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  17. Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, Yuxin Peng, and Zheng Zhang. Error-driven incremental learning in deep convolutional neural network for large-scale image classifi- cation. In Proceedings of the 22nd ACM international con- ference on Multimedia, pages 177-186. ACM, 2014.

Cited by

Sorted by citations

Authors
Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., & Fu, Y.
Affiliations
  • Northeastern University (US)
  • Microsoft (United States) (US)
  • City College of New York (US)
ORCID
Venue
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2019-06
Pages
374-382
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2019.00046
OpenAlex
W2954929116
Cited by
1,313
References
9
Citations by year
2026: 94 · 2025: 252 · 2024: 278 · 2023: 255 · 2022: 194 · 2021: 159 · 2020: 72 · 2019: 9
FWCI
41.51
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
catastrophic forgetting, incremental learning, class imbalance, large-scale classification, exemplar-based learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review