Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 16 of the 44 sources its claims cite were checked for support.

Mitigating catastrophic forgetting in lifelong learning: a hybrid architecture integrating neural ordinary differential equations with memory-augmented transformers

Open access
Zhou, Song, Li, Qiang · 2025 · Scientific Reports

Abstract

Catastrophic forgetting remains a critical challenge in lifelong learning where neural networks severely degrade on previous tasks when learning new ones. This paper proposes the first systematic integration of neural ordinary differential equations (Neural ODEs) with memory-augmented transformers to address this limitation. Neural ODEs enable smooth representation learning through continuous-time dynamics, while memory-augmented transformers provide explicit knowledge consolidation through attention-based retrieval. We derive rigorous PAC-learning theoretical bounds characterizing the relationships between model capacity, task sequence length, and forgetting severity, complemented by an adaptive memory management strategy. Comprehensive experiments on Split CIFAR-100, Permuted MNIST, and CORe50 demonstrate substantial improvements: 24% forgetting reduction and 10.3% accuracy gain over state-of-the-art methods. This work provides principled foundations for scalable lifelong learning systems capable of continuous knowledge accumulation without destructive interference.

Overview

CoChat Summary

## TL;DR

This paper proposes a hybrid continual-learning model that combines neural ordinary differential equations with memory-augmented transformers to reduce catastrophic forgetting. Across Split CIFAR-100, Permuted MNIST, and CORe50, it reports a 24% forgetting reduction and a 10.3% accuracy gain over the best baseline.

## Key findings

  • The proposed method achieved 72.6% accuracy on Split CIFAR-100 with a forgetting metric of 0.183, outperforming the best baseline (GEM) by 10.3% accuracy and 24.0% forgetting reduction.
  • Across the benchmarks Split CIFAR-100, Permuted MNIST, and CORe50, the paper reports substantial improvements in knowledge retention and learning performance relative to existing state-of-the-art continual learning methods.
  • Theoretical validation on Split CIFAR-100 with task sequences from 5 to 20 tasks showed observed forgetting rates remained below the derived upper bound by an average margin of 12.3%; for C = 15.8M parameters and n = 10 tasks, the bound predicted 0.217 maximum forgetting versus 0.189 empirically.
  • Ablation experiments report component contributions of 8.4% accuracy improvement from neural ODE layers, 6.7% from memory modules, and 4.2% from adaptive optimization; the full model reached 82.1% accuracy and 0.152 forgetting metric.

## Methods

  • The paper proposes a hybrid architecture that places neural ODE layers for continuous-time representation learning alongside memory-augmented transformer modules for attention-based knowledge retrieval and consolidation.
  • Experiments are run on Split CIFAR-100, Permuted MNIST, and CORe50, with comparisons against EWC, PackNet, GEM, and A-GEM; the reported environment uses PyTorch, NVIDIA A100 GPUs (40GB), batch size 64, learning rate 0.001, and an ODE solver (Dormand-Prince).
  • Analysis includes PAC-learning / VC-dimension-based theoretical bounds, forgetting metrics, backward transfer (BWT), forward transfer (FWT), ablation studies, and t-SNE / heatmap visualizations.

## Limitations

  • The paper states that computational overhead remains elevated, reporting 1.85× training time compared with standard fine-tuning.
  • The paper states that memory requirements grow linearly with task number, becoming unsustainable for very long sequences, specifically noting concern for > 500 tasks and elsewhere > 100 tasks for needing more advanced memory organization.
  • The paper states that the theoretical bounds can be loose in highly non-stationary scenarios and rely on assumptions such as Lipschitz continuity and bounded gradients, which may not hold universally.
  • Evident from the text: experimental validation is primarily on vision benchmarks, so generalization to NLP, audio, and multimodal settings is not demonstrated in the reported experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41598-025-31685-9

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Schematic illustration of the hybrid architecture integrating neural ODE layers, memory-augmented transformer modules, and multi-scale temporal processing pathways. The information flow initiates from input embeddings, propagates through continuous-time ODE blocks for smooth representation evolution, undergoes memory-augmented attention operations for knowledge consolidation, and ultimately produces task-specific outputs through hierarchical feature integration.
Fig. 1 Schematic illustration of the hybrid architecture integrating neural ODE layers, memory-augmented transformer modules, and multi-scale temporal processing pathways. The information flow initiates from input embeddings, propagates through continuous-time ODE blocks for smooth representation evolution, undergoes memory-augmented attention operations for knowledge consolidation, and ultimately produces task-specific outputs through hierarchical feature integration.CC BY-NC-ND · PubMed Central
Fig. 2 Illustration of the theoretical bounds on catastrophic forgetting as a function of model capacity, memory slots, and task sequence length. The surface plot demonstrates how forgetting rate decreases with increasing capacity while exhibiting logarithmic growth with task number, validating the derived theoretical predictions.
Fig. 2 Illustration of the theoretical bounds on catastrophic forgetting as a function of model capacity, memory slots, and task sequence length. The surface plot demonstrates how forgetting rate decreases with increasing capacity while exhibiting logarithmic growth with task number, validating the derived theoretical predictions.CC BY-NC-ND · PubMed Central
Fig. 3 Visualization of task sequences for the three benchmark datasets, illustrating the progression of learning challenges. Split CIFAR-100 shows class-incremental structure, Permuted MNIST demonstrates input transformation diversity, and CORe50 displays temporal object variations under changing environmental conditions.
Fig. 3 Visualization of task sequences for the three benchmark datasets, illustrating the progression of learning challenges. Split CIFAR-100 shows class-incremental structure, Permuted MNIST demonstrates input transformation diversity, and CORe50 displays temporal object variations under changing environmental conditions.CC BY-NC-ND · PubMed Central
Fig. 4 Enhanced flowchart depicting the experimental pipeline with four main stages: data preprocessing across three benchmark datasets, sequential task training protocol, baseline method comparisons, and comprehensive multi-metric evaluation. The color-coded stages (blue: preprocessing, green: training, orange: comparison, purple: evaluation) ensure visual clarity, while enlarged fonts and simplified structure facilitate understanding of the experimental workflow.
Fig. 4 Enhanced flowchart depicting the experimental pipeline with four main stages: data preprocessing across three benchmark datasets, sequential task training protocol, baseline method comparisons, and comprehensive multi-metric evaluation. The color-coded stages (blue: preprocessing, green: training, orange: comparison, purple: evaluation) ensure visual clarity, while enlarged fonts and simplified structure facilitate understanding of the experimental workflow.CC BY-NC-ND · PubMed Central
Fig. 5 Comparative performance visualization showing average accuracy and forgetting metrics for all evaluated methods across three benchmark datasets. The proposed method consistently outperforms baselines with significant margins, particularly in forgetting mitigation, approaching the theoretical upper bound established by joint training.
Fig. 5 Comparative performance visualization showing average accuracy and forgetting metrics for all evaluated methods across three benchmark datasets. The proposed method consistently outperforms baselines with significant margins, particularly in forgetting mitigation, approaching the theoretical upper bound established by joint training.CC BY-NC-ND · PubMed Central
Fig. 6 Visualization of ablation study demonstrating the incremental performance gains from each architectural component. The radar chart illustrates multi-dimensional improvements across accuracy, forgetting mitigation, forward transfer, and computational efficiency metrics.
Fig. 6 Visualization of ablation study demonstrating the incremental performance gains from each architectural component. The radar chart illustrates multi-dimensional improvements across accuracy, forgetting mitigation, forward transfer, and computational efficiency metrics.CC BY-NC-ND · PubMed Central
Fig. 7 t-SNE visualization of learned feature representations across sequential tasks. Different colors represent distinct tasks, demonstrating that the proposed architecture maintains well-separated task-specific clusters while forming coherent manifold structures. The gradual evolution of feature distributions illustrates smooth knowledge integration without abrupt geometric distortions characteristic of catastrophic forgetting.
Fig. 7 t-SNE visualization of learned feature representations across sequential tasks. Different colors represent distinct tasks, demonstrating that the proposed architecture maintains well-separated task-specific clusters while forming coherent manifold structures. The gradual evolution of feature distributions illustrates smooth knowledge integration without abrupt geometric distortions characteristic of catastrophic forgetting.CC BY-NC-ND · PubMed Central
Fig. 8 Heatmap visualization of the knowledge transfer matrix quantifying performance interactions between all task pairs. Diagonal elements represent individual task accuracies, while off-diagonal entries indicate transfer effects. Warm colors denote positive transfer, while cool colors indicate interference. The proposed method exhibits predominantly positive off-diagonal values, demonstrating beneficial knowledge reuse with minimal negative transfer.
Fig. 8 Heatmap visualization of the knowledge transfer matrix quantifying performance interactions between all task pairs. Diagonal elements represent individual task accuracies, while off-diagonal entries indicate transfer effects. Warm colors denote positive transfer, while cool colors indicate interference. The proposed method exhibits predominantly positive off-diagonal values, demonstrating beneficial knowledge reuse with minimal negative transfer.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

10 Valid · 8 Not valid · 27 Not checked · 41 Own results and interpretations

41 more claims · 4 with their cited sources checked

Create a free account to see all 86 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 29
  • Probably not a substantial claim. 9
  • The quote only partly states this claim. 7

Create a free account to see these claims.

Citations

Cites

Show all 32 references

Research works printed but not linked: 16

  1. Mundt, M., Lang, S., Delfosse, Q. & Kersting, K. CLEVA-Compass: a continual learning evaluation framework. Proc. Mach. Learn. Res. 202, 25124–25149 (2023).
  2. Yoon, J., Yang, E., Lee, J. & Hwang, S. J. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations (2018). https://openreview.net/forum?id=Sk7KsfW0-.
  3. Chaudhry, A., Ranzato, M., Rohrbach, M. & Elhoseiny, M. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations (2019). https://openreview.net/forum?id=Hkf2_sC5FX.
  4. Chen, R. T. Q., Rubanova, Y., Bettencourt, J. & Duvenaud, D. K. Neural ordinary differential equations. Adv. Neural Inf. Process. Syst. 31, 6572–6583 (2018).
  5. Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V. & Mishchenko, E. F. The Mathematical Theory of Optimal Processes (Wiley, 1962).
  6. Finlay, C., Jacobsen, J. H., Nurbekyan, L. & Oberman, A. M. How to train your neural ODE: the world of Jacobian and kinetic regularization. In Proceedings of the 37th International Conference on Machine Learning 3154–3164 (PMLR, 2020).
  7. Grathwohl, W., Chen, R. T. Q., Bettencourt, J., Sutskever, I. & Duvenaud, D. FFJORD: free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations (2019).
  8. Vaswani, A. et al. Attention is all you need. Adv. Neural. Inf. Process. Syst. 30, 5998–6008 (2017).
  9. Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D. & Lillicrap, T. Meta-learning with memory-augmented neural networks. In Proceedings of the 33rd International Conference on Machine Learning 1842–1850 (PMLR, 2016).
  10. Sukhbaatar, S., Szlam, A., Weston, J. & Fergus, R. End-to-end memory networks. In Advances in Neural Information Processing Systems 28 2440–2448 (2015).
  11. Jaegle, A. et al. Perceiver: General perception with iterative attention. In Proceedings of the 38th International Conference on Machine Learning 4651–4664 (PMLR, 2021).
  12. Kim, G., Xiao, C., Konishi, T., Ke, Z. & Liu, B. A theoretical study on solving continual learning. Adv. Neural Inform. Process. Syst. 36, 2563 (2024).
  13. Khalil, H. K. Nonlinear Systems 3rd edn (Prentice Hall, 2002).
  14. Borsos, Z., Mutny, M. & Krause, A. Coresets via bilevel optimization for continual learning and streaming. In Advances in Neural Information Processing Systems 33 14879–14890 (2020).
  15. Farajtabar, M., Azizan, N., Mott, A. & Li, A. Orthogonal gradient descent for continual learning. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics 3762–3773 (PMLR., 2020).
  16. Swaroop, S., Jain, C., Salakhutdinov, R. & Nguyen, C. Improving and understanding variational continual learning. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics 3337–3345 (PMLR, 2021).

Metadata

Authors
Zhou, S., & Li, Q.
Affiliations
Jiaxing Vocational Technical College (CN)
Corresponding author
Li, Qiang
Venue
Scientific Reports
Published
2025-12-15
Volume
16
Issue
1
Pages
2012-2012
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4417310185
Cited by
0
References
32
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
catastrophic forgetting, lifelong learning, neural ordinary differential equations, memory-augmented transformer, continuous-time dynamics, knowledge consolidation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Mitigating catastrophic forgetting in lifelong learning: a hybrid architecture integrating neural ordinary differential equations with memory-augmented transformers. https://app.cochat.ai/papers/10.1038/s41598-025-31685-9
The paper
Zhou, S., & Li, Q. (2025). Mitigating catastrophic forgetting in lifelong learning: a hybrid architecture integrating neural ordinary differential equations with memory-augmented transformers. Scientific Reports. https://doi.org/10.1038/s41598-025-31685-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.