CoChat
Sign Up
Cited by 27Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsReinforcement Learning in Robotics

Adversarial Feature Alignment: Avoid Catastrophic Forgetting in Incremental Task Lifelong Learning

Open access
Yao, Xin, Huang, Tianchi, Wu, Chenglei, Zhang, Rui-Xiao, Sun, Lifeng·2019·Neural Computation
DOI 10.1162/neco_a_01232
Open this paper in CoChat — free accountOpen PDF

Abstract

, is one of the major roadblocks that prevent deep neural networks from achieving human-level artificial intelligence. Several research efforts (e.g., lifelong or continual learning algorithms) have proposed to tackle this problem. However, they either suffer from an accumulating drop in performance as the task sequence grows longer, or require storing an excessive number of model parameters for historical memory, or cannot obtain competitive performance on the new tasks. In this letter, we focus on the incremental multitask image classification scenario. Inspired by the learning process of students, who usually decompose complex tasks into easier goals, we propose an adversarial feature alignment method to avoid catastrophic forgetting. In our design, both the low-level visual features and high-level semantic features serve as soft targets and guide the training process in multiple stages, which provide sufficient supervised information of the old tasks and help to reduce forgetting. Due to the knowledge distillation and regularization phenomena, the proposed method gains even better performance than fine-tuning on the new tasks, which makes it stand out from other methods. Extensive experiments in several typical lifelong learning scenarios demonstrate that our method outperforms the state-of-the-art methods in both accuracy on new tasks and performance preservation on old tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/neco_a_01232

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Compared with LwF, AFA improves the distillation loss and increases the restraint of the model, but it does not significantly improve the performance of the model.

    “Compared with LwF, AFA improves the distillation loss and increases the restraint of the model, but it does not significantly improve the performance of the model.”

    From An Appraisal of Incremental Learning Methods (2020) · cites this paper as [71]

  • Many distillation strategy methods can only effectively mitigate catastrophic forgetting when the number of incremental tasks is small, and still difficult to meet long-term increments.

    “Many distillation strategy methods can only effectively mitigate catastrophic forgetting when the number of incremental tasks is small, in which it is still difficult to meet the requirements of long-term increments [16,70,71].”

    From An Appraisal of Incremental Learning Methods (2020) · cites this paper as [71]

  • Using similar experimental settings, the gap between results of three distillation-based methods is not obvious; this means soft label supervision has not been significantly enhanced, and using only new data is not reliable at present.

    “Using similar experimental settings, the gap between the results of the three methods based on the distillation strategy is not too obvious [16,70,71]. This means that the supervision ability of soft labels has not been significantly enhanced, i.e., the method of using only new data is not reliable at present.”

    From An Appraisal of Incremental Learning Methods (2020) · cites this paper as [71]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 18

  1. [4][4] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research , 17(1):2096-2030, 2016. 6 -
  2. [5][5] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256, 2010. 7 -
  3. [6][6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680, 2014. 3 -
  4. [8][8] A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal kernel choice for large-scale twosample tests. In Advances in neural information processing systems , pages 1205-1213, 2012. 2, 6 -
  5. [12][12] R. Kemker, M. McClure, A. Abitino, T. L. Hayes, and C. Kanan. Measuring catastrophic forgetting in neural networks. In Thirty-second AAAI conference on artificial intelligence , 2018. 2 -
  6. [16][16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012. 5, 6 -
  7. [19][19] V. Lomonaco and D. Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning , pages 17-26, 2017. 2, 3, 4 -
  8. [20][20] M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning , pages 97-105, 2015. 6, 11 -
  9. [21][21] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning , pages 2208-2217, 2017. 11 -
  10. [22][22] D. Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems , pages 6467-6476, 2017. 2, 3, 4 -
  11. [24][24] D. Maltoni and V. Lomonaco. Continuous learning in single-incremental-task scenarios. arXiv preprint arXiv:1806.08568 , 2018. 2 - · arXiv
  12. [28][28] A. Rannen Ep Triki, R. Aljundi, M. Blaschko, and T. Tuytelaars. Encoder based lifelong learning. In Proceedings ICCV 2017 , pages 1320-1328, 2017. 3, 4, 6, 7 -
  13. [30][30] A. Rosenfeld and J. K. Tsotsos. Incremental learning through deep adaptation. arXiv preprint arXiv:1705.04228 , 2017. 2, 3 - · arXiv
  14. [32][32] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2994-3003. Curran Associates Inc., 2017. 3, 4 -
  15. [33][33] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , pages 1139-1147, 2013. 7 -
  16. [37][37] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems , pages 3320-3328, 2014. 6 -
  17. [38][38] S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International conference on learning representations (ICLR) , 2017. 5 -
  18. [39][39] F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML 2017 , pages 3987-3995, 2017. 3, 4, 7, 8, 9

Cited by

Sorted by citations

Related

Authors
Yao, X., Huang, T., Wu, C., Zhang, R.-X., & Sun, L.
Affiliations
Tsinghua University (CN)
Venue
Neural Computation
Publisher
MIT Press - Journals
Published
2019-11
Volume
31
Issue
11
Pages
2266-2291
ISSN
  • 0899-7667 (ISSN-L)
  • 1530-888X
DOI
10.1162/neco_a_01232
OpenAlex
W2973324789
Cited by
27
References
13
Citations by year
2026: 3 · 2025: 4 · 2024: 5 · 2023: 1 · 2022: 4 · 2021: 6 · 2020: 4
FWCI
1.60
Type
journal article
Language
en
Open access
Locations
  • Neural Computation (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
catastrophic forgetting, lifelong learning, continual learning, image classification, knowledge distillation, soft targets, fine-tuning
MeSH terms
Brain, Humans, Learning, Models, Neurological, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review