The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Continual learning of context-dependent processing in neural networks
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that input created from a generative model can indeed assist with learning, such as preventing catastrophic forgetting.149”
From The overfitted brain: Dreams evolved to assist generalization (2021) · cites this paper as [149]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803
Neural Mechanisms of Selective Visual Attention
Desimone, R., & Duncan, J. (1995) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.18.030195.001205 · Cited by 8,388
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Research works printed but not linked: 15
- Newell, A. Unified Theories of Cognition (Harvard Univ. Press, 1994).
- Haykin, S. S. Adaptive Filter theory (Pearson Education India, 2008).
- Singhal, S. & Wu, L. Training feed-forward networks with the extended kalman algorithm. In International Conference on Acoustics, Speech, and Signal Processing 1187–1190 (IEEE, 1989).
- Jaeger, H. Controlling recurrent neural networks by conceptors. Preprint at https://arxiv.org/abs/1403.3369 (2014). · arXiv
- Nair, V. & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning 807–814 (PMLR, 2010).
- Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W. & Zhang, B.-T. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems 4652–4662 (Curran Associates, 2017).
- Zenke, F., Poole, B. & Ganguli, S. Continual learning through synaptic intelligence. In International Conference on Machine Learning 6072–6082 (PMLR, 2017).
- Řehůřek, R. & Sojka, P. Software framework for topic modelling with large corpora. Proc. LREC 2010 Workshop on New Challenges for NLP Frameworks 45–50 (ELRA, 2010).
- Shin, H., Lee, J. K., Kim, J. & Kim, J. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems 2990–2999 (Curran Associates, 2017).
- Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 3320–3328 (Curran Associates, 2014).
- Hinton, G., Vinyals, O. & Dean, J. Distilling the knowledge in a neural network. Preprint at https://arxiv.org/abs/1503.02531 (2015). · arXiv
- Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. Thirteenth International Conference on Artificial Intelligence and Statistics 249–256 (Microtome, 2010).
- Nair, V. & Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning (ICML-10) 807–814 (PMLR, 2010).
- Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F. & Schmidhuber, J. Compete to compute. In Advances in Neural Information Processing Systems 2310–2318 (Curran Associates, 2013).
- Hu, W. et al. Overcoming catastrophic forgetting via model adaptation. In International Conference on Learning Representations (ICLR, 2019).
Cited by
Sorted by citations
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
“Alternatively, gradient projection can also be performed without storing old training samples [51], [65], [66], [67].”
BackgroundThree types of incremental learning
van de Ven, G. M., Tuytelaars, T., & Tolias, A. S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00568-3 · Cited by 665
“2 shows how Split MNIST, which is a popular toy problem for continual learning 27,28,31,32 , can be performed according to each of the three scenarios.”
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397
“Hebbian learning might encourage the formation of orthogonal neural codes for different temporal contexts 133 , which in turn enables tasks to be learned in different neural subspaces 132 .”
BackgroundClass-Incremental Learning: A Survey
Zhou, D.-W., Wang, Q.-W., Qi, Z.-H., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429383 · Cited by 376
“OWM [56] only allows weight modification in the direction orthogonal to the subspace spanned by all previously learned inputs.”
BackgroundInfluentialOrthogonal Gradient Descent for Continual LearningPreprint
Farajtabar, M., Azizan, N., Mott, A., & Li, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07104 · Cited by 184
Robust Spike-Based Continual Meta-Learning Improved by Restricted Minimum Error Entropy Criterion
Yang, S., Tan, J., & Chen, B. (2022) · Entropy · doi:10.3390/e24040455 · Cited by 157
“The human brain has implemented an efficient and scalable mechanism for continual learning based on neuronal activity patterns that represent previous experiences [6].”
BackgroundTraining Networks in Null Space of Feature Covariance for Continual Learning
Wang, S., Li, X., Sun, J., & Xu, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00025 · Cited by 135
“Different from GEM and A-GEM, OWM [54] projects the parameter up-date into the orthogonal space of the space spanned by input features of each linear layer.”
BackgroundMethodInfluentialMgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning
Zhao, H., Fu, Y., Kang, M., Tian, Q., Wu, F., & Li, X. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3133897 · Cited by 115
“Recently, there has been a large body of research in incremental learning [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43].”
A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
Wang, Z., Yang, E., Shen, L., & Huang, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3498346 · Cited by 132
“(2) Orthogonal Feature Subspace: These require that the parameter update direction of the new task is orthogonal to the subspace spanned by the input(feature) of the old tasks [62], [63], [64], [65], [66].”
The Computational and Neural Bases of Context-Dependent Learning
Heald, J. B., Wolpert, D. M., & Lengyel, M. (2023) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092322-100402 · Cited by 86
A framework for the general design and computation of hybrid neural networks
Zhao, R. J., Yang, Z., Zheng, H. R., Wu, Y., Liu, F., Wu, Z., Li, L. T., Chen, F., Song, S., Zhu, J., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30964-7 · Cited by 80
GAN Memory with No ForgettingPreprint
Cong, Y., Zhao, M., Li, J., Wang, S., & Carin, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07543 · Cited by 77
“To alleviate catastrophic forgetting, many methods have been proposed, with most focusing on discriminative/classification tasks [36, 65, 95, 55, 94].”
MethodZhang, T., Cheng, X., Jia, S., Poo, M., Zeng, Y., & Xu, B. (2021) · Science Advances · doi:10.1126/sciadv.abh0146 · Cited by 78
“E j SNN = a V j (t) 2 + b ∑ i N ( V i (t ) W i,j V j (t ) ) + c V j (t) (6)”
The overfitted brain: Dreams evolved to assist generalization
Hoel, E. (2021) · Patterns · doi:10.1016/j.patter.2021.100244 · Cited by 70
“It has been shown that input created from a generative model can indeed assist with learning, such as preventing catastrophic forgetting.(149) It should be noted that in most these cases the generative model exists outside the network itself, which is not biologically realistic in the case of the brain, although there…”
BackgroundContinual Learning: A Review of Techniques, Challenges, and Future Directions
Wickramasinghe, B., Saha, G., & Roy, K. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3339091 · Cited by 82
“Orthogonal weight modulation (OWM) reduces interference to old information by modifying the weights of the network in a direction orthogonal to the subspace spanned by the inputs of old tasks [142].”
BackgroundA Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials
Li, C., Huang, Z., Paudel, D. P., Wang, Y., Shahbazi, M., Hong, X., & Gool, L. V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00139 · Cited by 76
“1 Gradient-based methods (e.g., [62, 85, 23, 65, 70, 75]) overcome catastrophic forgetting by minimizing the interference among task-wise gradients when updating network weights.”
MethodRecent Advances of Continual Learning in Computer Vision: An Overview
Qu, H., Rahmani, H., Xu, L., Williams, B. M., & Liu, J. (2025) · IET Computer Vision · doi:10.1049/cvi2.70013 · Cited by 122
Lifelong Incremental Reinforcement Learning With Online Bayesian Inference
Wang, Z., Chen, C., & Dong, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3055499 · Cited by 63
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector
Lin, G., Chu, H., & Lai, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00019 · Cited by 67
“OWM [43] is orthogonal weight modification method to overcome catastrophic forgetting.”
An Intelligent Transient Stability Assessment Framework with Continual Learning Ability
Li, X., Yang, Z., Guo, P., & Cheng, J. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3064052 · Cited by 62
Continual All-in-One Adverse Weather Removal With Knowledge Replay on a Unified Network Structure
Cheng, D., Ji, Y., Gong, D., Li, Y., Wang, N., Han, J., & Zhang, D. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3377136 · Cited by 60
“Some others, like Adam-NSC [33] and OWM [34], regulate the direction of parameter change.”
BackgroundTuning Convolutional Spiking Neural Network With Biologically Plausible Reward Propagation
Zhang, T., Jia, S., Cheng, X., & Xu, B. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3085966 · Cited by 62
Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks
Tadros, T., Krishnan, G. P., Ramyaa, R., & Bazhenov, M. (2022) · Nature Communications · doi:10.1038/s41467-022-34938-7 · Cited by 62
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks
Ke, Z., Xu, H., & Liu, B. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.378 · Cited by 70
“HAT, UCL, EWC and OWM were originally designed for image classification.”
MethodInfluentialContinual learning and catastrophic forgetting
van de Ven, G. M., Soures, N., & Kudithipudi, D. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-15754-7.00073-0 · Cited by 165
“A first popular approach is ‘orthogonal gradient projection’ (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021).”
BackgroundCatastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
“It is supported by finding orthogonal filters for each task and, in inference time, using the task identifier to retrieve the right filters [Zeng et al ., 2019].”
MethodChen, J., & Qiao, H. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.3045574 · Cited by 52
“To realize continuous multitask learning with RNN, the OWM method [60] is introduced and improved in this section.”
MethodHuang, X., Xie, T., Luo, S., Wu, J., Luo, R., & Zhou, Q. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108212 · Cited by 53
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54
Flesch, T., Nagy, D., Saxe, A., & Summerfield, C. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010808 · Cited by 53
“Orthogonalisation approaches in particular have gained attention in neuroscience [31, 33], as more recent investigations of neural geometry have shown that during multi-task performance, mutual interference among tasks is minimised by projecting relevant dimensions into orthogonal, low-dimensional subspaces [15, 17, 4…”
BackgroundCLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks
Ke, Z., Liu, B., Xu, H., & Shu, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.550 · Cited by 51
Organizing recurrent network dynamics by task-computation to enable continual learning
Duncker, L., Driscoll, L. N., Shenoy, K. V., Sahani, M., & Sussillo, D. (2020) · UCL Discovery (University College London) · Cited by 42
Brain works principle followed by neural information processing: a review of novel brain theory
Wang, R., Wang, Y., Xu, X., Li, Y., & Pan, X. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10520-5 · Cited by 43
Electrochemical ohmic memristors for continual learning
Chen, S., Yang, Z., Hartmann, H., Besmehn, A., Yang, Y., & Valov, I. (2025) · Nature Communications · doi:10.1038/s41467-025-57543-w · Cited by 49
“5a) 45 .”
Linear Mode Connectivity in Multitask and Continual LearningPreprint
Mirzadeh, S. I., Farajtabar, M., Görür, D., Pascanu, R., & Ghasemzadeh, H. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04495 · Cited by 55
Sun, W., Li, Q., Zhang, J., Wang, D., Wang, W., & Geng, Y. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109561 · Cited by 44
Memory Recall: A Simple Neural Network Training Framework Against Catastrophic Forgetting
Zhang, B., Guo, Y., Li, Y., He, Y., Wang, H., & Dai, Q. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099700 · Cited by 42
A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold
Fei, Y., Liu, Y., Jia, C., Zheng-yu, L., Wei, X. H., & Chen, M. (2024) · ACM Computing Surveys · doi:10.1145/3708498 · Cited by 50
“of gradient orthogonality was first proposed in [90].”
Hardware Acceleration of EEG-Based Emotion Classification Systems: A Comprehensive Survey
Gonzalez, H. A., George, R., Muzaffar, S., Acevedo, J., Höppner, S., Mayr, C., Yoo, J., Fitzek, F. H. P., & Elfadel, I. M. (2021) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2021.3089132 · Cited by 39
“Another learning paradigm that is also geared toward improving model performance as old data is updated or new data becomes available is task-progressive [212], continual [213], or lifelong learning [214].”
BackgroundCKDF: Cascaded Knowledge Distillation Framework for Robust Incremental Learning
Li, K., Wan, J., & Yu, S. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3176130 · Cited by 36
“1) Regularization Approaches: The first is regularization approaches [14]–[20].”
BackgroundZhang, T., Cheng, X., Jia, S., Li, C., Poo, M., & Xu, B. (2023) · Science Advances · doi:10.1126/sciadv.adi2947 · Cited by 40
CRNet: A Fast Continual Learning Framework With Random Theory
Li, D., & Zeng, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3262853 · Cited by 39
“We follow the setting in OWM [56] where a feature extractor pretrained by the complete training sets is first used to analyze the”
MethodInfluentialContinual task learning in natural and artificial agents
Flesch, T., Saxe, A., & Summerfield, C. (2023) · Trends in Neurosciences · doi:10.1016/j.tins.2022.12.006 · Cited by 41
“As a proof of concept, methods that project learning updates (gradients) for new tasks into independent neural subspaces have been shown to help with continual learning in machine learning models, including recurrent networks [67,68].”
Variational Data-Free Knowledge Distillation for Continual Learning
Li, X., Wang, S., Sun, J. Z., & Xu, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3271626 · Cited by 34
One-shot Federated Learning without server-side training
Su, S., Li, B., & Xue, X. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.04.035 · Cited by 34
“The projection matrix has been applied into Continual Learning [40, 41, 42].”
MethodRegularizing Second-Order Influences for Continual Learning
Sun, Z., Mu, Y., & Hua, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01931 · Cited by 35
“To overcome this effect, three main streams of methods have been developed: weight regularization [28, 33, 57], parameter isolation [2, 43, 45, 56] and memory replay [34, 41, 46].”
Domain-incremental learning for fire detection in space-air-ground integrated observation network
Wang, M., Yu, D., He, W., Yue, P., & Liang, Z. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103279 · Cited by 33
Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning
Kong, Y., Liu, L., Wang, Z., & Tao, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_13 · Cited by 32
Embodied tactile perception and learning
Liu, H., Guo, D., Sun, F., Yang, W., Furber, S. B., & Sun, T. (2020) · Brain Science Advances · doi:10.26599/bsa.2020.9050012 · Cited by 32
“The team demonstrated a good lifelong learning ability in applications such as handwriting and face recognitions [134].”
BackgroundWhat to remember: Self-adaptive continual learning for audio deepfake detection
Zhang, X., Yi, J., Wang, C., Zhang, C. Y., Zeng, S., & Tao, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29929 · Cited by 32
Neural Manifold Modulated Continual Reinforcement Learning for Musculoskeletal Robots
Chen, J., Chen, Z., Yao, C., & Qiao, H. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3231055 · Cited by 28
“For projection constraints-based methods, the input space of each layer in network is constructed by different ways and the update of weights is projected to the direction orthogonal to the input space [26], [27].”
BackgroundMethodHua, J., Li, Y., Mou, W., & Liu, C. (2021) · Proceedings of the Institution of Mechanical Engineers Part B Journal of Engineering Manufacture · doi:10.1177/0954405421993694 · Cited by 27
“24,25 To avoid catastrophic forgetting, the orthogonal weights modification method 26 is used for continual learning.”
MethodSpiking neural predictive coding for continually learning from data streams
Ororbia, A. G. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126292 · Cited by 43
Continual Learning with Scaled Gradient Projection
Saha, G., & Roy, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26157 · Cited by 28
Kumari, P., Chauhan, J., Bozorgpour, A., Huang, B., Azad, R., & Merhof, D. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103730 · Cited by 46
Rethinking Gradient Projection Continual Learning: Stability/Plasticity Feature Space Decoupling
Zhao, Z. W., Zhang, Z., Tan, X., Liu, J., Qu, Y., Xie, Y., & Ma, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00362 · Cited by 29
A Dual-Channel Collaborative Transformer for continual learning
Cai, H., Wang, Y., Luo, Y., & Mao, K. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.112792 · Cited by 26
Continual Learning with Knowledge Transfer for Sentiment Classification
Ke, Z., Liu, B., Wang, H., & Shu, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67664-3_41 · Cited by 26
“Orthogonal Weights Modification (OWM) [32] is a state-of-the-art class continual learning (CCL) method.”
BackgroundMethodInfluentialDu, J., Liu, P., Vong, C., Chen, C., Wang, T., & Chen, C. L. P. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3259016 · Cited by 24
“…and it is also compared with the existing DCIL methods, including EWC (2017) [33], LwF (2017) [34], incremental moment matching (IMM) (2017) [47], OWM (2019) [35], PGMA (2019) [36], random patch selection network (RPSnet) (2019) [48], and PCL (2021) [18] over five real-world image datasets.”
MethodLifelong Learning With Cycle Memory Networks
Peng, J., Ye, D., Tang, B., Lei, Y., Liu, Y., & Li, H. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294495 · Cited by 25
“Zeng et al. [46] and Tang et al. [47] kept the parameter space of the preceding and following tasks orthogonal to avoid interference.”
BackgroundNikolić, D. (2023) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2023.107820 · Cited by 27
Understanding neural flexibility from a multifaceted definition
Yin, D., & Kaiser, M. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118027 · Cited by 24
“recently proposed an approach that includes an orthogonal weight modification algorithm and a context-dependent processing module, enabling a description of a neural network to progressively learn various mapping rules in a context-dependent manner (Zeng et al., 2019).”
Disturbance-immune weight sharing for neural architecture search
Niu, S., Wu, J., Zhang, Y., Guo, Y. F., Zhao, P., Huang, J., & Tan, M. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.002 · Cited by 23
Verma, T., Jin, L., Zhou, J. W., Huang, J., Tan, M., Choong, B. C. M., Tan, T. F., Gao, F., Xu, X., Ting, D. S. W., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1227515 · Cited by 25
Ma, L., Zhou, Y., Ma, Y., Yu, G., Li, Q., He, Q., & Pei, Y. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3540650 · Cited by 22
“Unfortunately, the projector attenuation will ultimately and severely degrade the efficiency [24] of the weight learning as well as the prediction ability of the supernet.”
Kang, M., Zhang, J., Zhang, J., Wang, X., Yang, C., Ma, Z., & Huang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01732 · Cited by 23
Generative Feature Replay with Orthogonal Weight Modification for Continual Learning
Shen, G., Zhang, S., Chen, X., & Deng, Z. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534437 · Cited by 25
Remaining Useful Life Prediction Based on Incremental Learning
Que, Z., Jin, X., Xu, Z., & Hu, C. (2023) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3294939 · Cited by 23
“The OWM algorithm [27], as shown in Fig.”
MethodFedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning
Lee, G., Jeong, M., Kim, S., Oh, J., & Yun, S.-Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01189 · Cited by 22
“Second, unlike in CL where task distributions are often disjoint [13, 72], FL clients may have overlapping data distributions, including instances from the same class.”
BackgroundDealing with Cross-Task Class Discrimination in Online Continual Learning
Guo, Y., Liu, B., & Zhao, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01143 · Cited by 21
Brain-Inspired Collaborative Power Dynamic Dispatch for the Digital Twin Smart Generation System
Liu, Z., Xi, L., Quan, Y., & Cheng, C. (2024) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2024.3373371 · Cited by 21
Rapid context inference in a thalamocortical model using recurrent neural networks
Zheng, W., Wu, Z., Hummos, A., Yang, G. R., & Halassa, M. M. (2024) · Nature Communications · doi:10.1038/s41467-024-52289-3 · Cited by 22
Recursive Least-Squares Estimator-Aided Online Learning for Visual Tracking
Gao, J. G., Hu, W., & Lu, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00741 · Cited by 21
“It has recently proved valuable in numerous supervised learning and reinforcement learning-based applications along with deep learning success, leading to an emerging field [18, 15, 39, 27].”
BackgroundContinual learning in medical image analysis: A survey
Wu, X., Xu, Z., & Tong, R. K. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109206 · Cited by 29
Learning Without Forgetting: A New Framework for Network Cyber Security Threat Detection
Karn, R. R., Kudva, P. N., & Elfadel, I. M. (2021) · IEEE Access · doi:10.1109/access.2021.3115946 · Cited by 25
“The recommended method to compute O µ + 1 is through an iterative process as in the Recursive Least Squares algo-rithm [6], with O µ + 1 computed for each layer separately using only the training dataset D µ + 1 and the projection for the most recent task O µ .”
BackgroundMethodInfluentialOpen-world continual learning: Unifying novelty detection and continual learning
Kim, G., Xiao, C., Konishi, T., Ke, Z., & Liu, B. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2024.104237 · Cited by 19
Continual Learning Based on OOD Detection and Task Masking
Kim, G., Esmaeilpour, S., Xiao, C., & Liu, B. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00431 · Cited by 19
Multi-view class incremental learning
Li, D., Wang, T., Chen, J., Kawaguchi, K., Lian, C., & Zeng, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102021 · Cited by 20
“To make the comparison more complete, we also compare with CIL algorithms: (1) EWC [45]; (2) PCL [80]; (3) OWM [62]; (4) BiC [81]; (5) FS-DGPM [82]; (6) GEM [83]; (7) LOGD [84]; (8) IL2M [41].”
BackgroundMethodInfluentialExamining Adaptive E-Learning Approaches to Enhance Learning and Individual Experiences
Benkhalfallah, F., Laouar, M. R., & Benkhalfallah, M. S. (2024) · Acta Informatica Pragensia · doi:10.18267/j.aip.240 · Cited by 19
Energy-Based Models for Continual LearningPreprint
Li, S., Du, Y., van de Ven, G. M., & Mordatch, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12216 · Cited by 19
“, 2018) or by using a different set of parameters for each task (Zeng et al., 2019; Hu et al., 2019).”
BackgroundNatural continual learning: success is a journey, not (just) a destination
Kao, T.-C., Jensen, K. T., van de Ven, G. M., Bernacchia, A., & Hennequin, G. (2021) · Lirias · doi:10.48550/arxiv.2106.08085 · Cited by 18
“…projection matrix P L designed to approximate the posterior covariance of previous tasks Λ − 1 k − 1 ≈ P R ⊗ P L (i.e., the prior covariance on task k ; Appendix D), while Zeng et al. (2019) use the identity matrix I and Duncker et al. (2020) use the covariance of recurrent inputs (Appendix F).”
BackgroundMethodInfluentialMemory efficient data-free distillation for continual learning
Li, X., Wang, S., Sun, J., & Xu, Z. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109875 · Cited by 18
Data Augmented Flatness-aware Gradient Projection for Continual Learning
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., & Wang, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00518 · Cited by 18
“To alleviate catastrophic forgetting of old tasks, several works [57, 15, 8, 27, 43] propose to constrain the gradient update direction of the new tasks.”
BackgroundInfluentialLi, K., Chen, H., Wan, J., & Yu, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376197 · Cited by 20
“Regularization-based approaches introduce different regularization terms in loss functions to penalize the changes of network outputs to alleviate forgetting [9], [10].”
BackgroundSPACE: Structured Compression and Sharing of Representational Space for Continual Learning
Saha, G., Garg, I., Ankit, A., & Roy, K. (2021) · IEEE Access · doi:10.1109/access.2021.3126027 · Cited by 22
“However, in the context of continual learning (CL) [3, 4, 5, 6, 7], whereas humans can continuously adapt and learn new tasks throughout their lifetimes without forgetting the old ones, current deep learning methods struggle to perform well since the data distribution changes over the course of learning.”
BackgroundBeneficial Perturbation Network for Designing General Adaptive Artificial Intelligence Systems
Wen, S., Rios, A., Ge, Y., & Itti, L. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3054423 · Cited by 18
Hierarchically structured task-agnostic continual learning
Hihn, H., & Braun, D. A. (2022) · Machine Learning · doi:10.1007/s10994-022-06283-9 · Cited by 20
“, by mapping the gradient updates into a different space [26, 27].”
BackgroundHan, S., Liu, B., Fan, X., Feng, T., Yang, J., Zhou, Z., Gong, H., & Luo, J. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105759 · Cited by 17
Signatures of task learning in neural representations
Gurnani, H., & Cayco-Gajic, N. A. (2023) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2023.102759 · Cited by 19
GopGAN: Gradients Orthogonal Projection Generative Adversarial Network With Continual Learning
Li, X., & Wang, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3093319 · Cited by 16
“The pioneering works, CAB [10] and OWM [39], are the variants of the backpropagation algorithm and have been successfully applied in classification tasks, but we find that there is no work to apply these two methods to GANs.”
MethodInfluentialContinual learning: A systematic literature review
Yang, Q., Wang, L., Wicker, J., & Dobbie, G. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108226 · Cited by 23
Class Gradient Projection For Continual Learning
Chen, C., Zhang, J., Song, J., & Gao, L. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548054 · Cited by 15
“Recently, the gradient projection based paradigms [14, 35, 44] have achieved remarkable performance in tackling the catastrophic forgetting problem in CL.”
BackgroundMethodZheng, H., Liu, B., Han, S., Fan, X., Zou, T., Zhou, Z., & Gong, H. (2022) · Environmental Earth Sciences · doi:10.1007/s12665-022-10369-x · Cited by 15
A Two-Level Dynamic Adaptive Network for Medical Image Fusion
Huang, W., Zhang, H., Quan, X., & Wang, J. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3169546 · Cited by 15
LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning
Liu, X., & Chang, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01426 · Cited by 17
“Existing works [31, 37, 45] have shown that the gradient update in a linear layer lies within the span of the input data.”
Research on landslide susceptibility prediction model based on LSTM-RF-MDBN
Yang, X., Fan, X., Wang, K., & Zhou, Z. (2023) · Environmental Science and Pollution Research · doi:10.1007/s11356-023-31232-x · Cited by 14
Holton, E., Braun, L., Thompson, J. A. F., Grohn, J., & Summerfield, C. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02318-y · Cited by 18
“This aligns with a growing literature in machine learning showing that catastrophic forgetting can be mitigated by encouraging networks to learn in orthogonal subspaces in more complex settings 45–49 .”
Adaptive Domain-Aware Representation Learning for Speech Emotion Recognition
Fan, W., Xu, X., Xing, X., & Huang, D. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2572 · Cited by 14
“Inspired by [20], we propose an adaptive domain-aware representation Learning (ADARL) in this paper, which consists of a multi-task representation learning by combining domain clas-sification and emotion recognition, and an extracting domain aware representation through the domain knowledge…”
BackgroundMethodCognitively Inspired Learning of Incremental Drifting Concepts
Rostami, M. G., & Galstyan, A. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/341 · Cited by 14
incDFM: Incremental Deep Feature Modeling for Continual Novelty Detection
Rios, A., Ahuja, N., Ndiour, I. J., Genc, U., Itti, L., & Tickoo, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19806-9_34 · Cited by 13
“…of task labels which are usually provided by a task oracle and utilize this information by employing task-specific classification heads and other task-dependent parameters to share the rest of the network for different tasks, e.g. partitioning with context [48, 47, 5] or mask matrices [27, 9].”
Method
Metadata
- Authors
- Zeng, G., Chen, Y., Cui, B., & Yu, S.
- Affiliations
- Chinese Academy of Sciences (CN)
- ORCID
- Yu, Shan
- Venue
- Nature Machine Intelligence
- Published
- 2019-08-09
- Volume
- 1
- Issue
- 8
- Pages
- 364-372
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W2895699061
- Cited by
- 309
- References
- 41
- Citations by year
- 2026: 51 · 2025: 63 · 2024: 54 · 2023: 52 · 2022: 23 · 2021: 39 · 2020: 21 · 2019: 6
- FWCI
- 10.27
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Nature Machine Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- continual learning, catastrophic forgetting, context-dependent processing, neural network modules, multi-task learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
