The DOI matches its DataCite record; 32 of the 34 references it cites were found in a registry or an index.
Overcoming catastrophic forgetting with hard attention to the task
Open accessAbstract
Catastrophic forgetting occurs when a neural network loses the information learned in a previous task after training on subsequent tasks. This problem remains a hurdle for artificial intelligence systems with sequential learning capabilities. In this paper, we propose a task-based hard attention mechanism that preserves previous tasks' information without affecting the current task's learning. A hard attention mask is learned concurrently to every task, through stochastic gradient descent, and previous masks are exploited to condition such learning. We show that the proposed mechanism is effective for reducing catastrophic forgetting, cutting current rates by 45 to 80%. We also show that it is robust to different hyperparameter choices, and that it offers a number of monitoring capabilities. The approach features the possibility to control both the stability and compactness of the learned knowledge, which we believe makes it also attractive for online learning or network compression applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Methods like regularizing changes to prevent overwriting important weights and applying soft masks to selectively update parameters can largely preserve the previous domain knowledge while recognizing new information (Serra et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Serra et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
(2017) · Cited by 11,625
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Regularized multi--task learning
Evgeniou, T., & Pontil, M. (2004) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1014052.1014067 · Cited by 1,473
Continual Learning with Deep Generative Replay
Shin, H., Lee, J. K., Kim, J., & Kim, J. (2017) · arXiv · doi:10.48550/arxiv.1705.08690 · Cited by 1,385
PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
Mallya, A., & Lazebnik, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00810 · Cited by 1,258
Lifelong Learning with Dynamically Expandable NetworksPreprint
Yoon, J., Yang, E., Lee, J., & Hwang, S. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01547 · Cited by 1,090
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089
PathNet: Evolution Channels Gradient Descent in Super Neural NetworksPreprint
Fernando, C., Banarse, D., Blundell, C., Zwólš, Y., Ha, D., Rusu, A. A., Pritzel, A., & Wierstra, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.08734 · Cited by 966
The MegaFace Benchmark: 1 Million Faces for Recognition at Scale
Kemelmacher-Shlizerman, I., Seitz, S. M., Miller, D., & Brossard, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.527 · Cited by 912
Catastrophic forgetting, rehearsal and pseudorehearsal
ROBINS, A. (1995) · Connection Science · doi:10.1080/09540099550039318 · Cited by 833
A data-driven approach to cleaning large face datasets
Ng, H., & WINKLER, S. S. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025068 · Cited by 673
Li, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659
Thrun, S., & Mitchell, T. M. (1995) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(95)00004-y · Cited by 536
Overcoming Catastrophic Forgetting by Incremental Moment MatchingPreprint
Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., & Zhang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08475 · Cited by 498
Variational Continual Learning
Turner, R., Bui, T., Li, Y., & Cuong, N. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.35471 · Cited by 438
On the Origin of Deep LearningPreprint
Wang, H., & Raj, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07800 · Cited by 312
(2000) · Applied Physics Letters · doi:10.1162/153244304322765658 · Cited by 192
Less-forgetting Learning in Deep Neural NetworksPreprint
Jung, H., Ju, J., Jung, M., & Kim, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.00122 · Cited by 160
Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F. J., & Schmidhuber, J. (2013) · neural information processing systems · Cited by 151
Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionist Networks
French, R. M. (1991) · Cited by 130
Overcoming Catastrophic Interference using Conceptor-Aided Backpropagation
He, X., & Jaeger, H. (2018) · International Conference on Learning Representations · Cited by 78
Memory-based Parameter AdaptationPreprint
Sprechmann, P., Jayakumar, S. M., Rae, J. W., Pritzel, A., Badia, A. P., Uría, B., Vinyals, O., Hassabis, D., Pascanu, R., & Blundell, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10542 · Cited by 54
Improved multitask learning through synaptic intelligence.Preprint
Zenke, F., Poole, B., & Ganguli, S. (2017) · arXiv (Cornell University) · Cited by 29
Reduction of catastrophic forgetting with transfer learning and ternary output codes
Gutstein, S., & Stump, E. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280416 · Cited by 14
Overcoming Catastrophic Interference by ConceptorsPreprint
He, X., & Jaeger, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04853 · Cited by 12
Research works printed but not linked: 13
- Bakker, B. and Heskes, T. Task clustering and gating for bayesian multitask learning. Journal of Machine Learn- ing Research, 4:83-99, 2003.
- Clegg, B. A., DiGirolamo, G. J., and Keele, S. W. Sequence learning. Trends in Cognitive Sciences, 2(8):275-281, 1998.
- Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. of the Int. Conf. on Artificial Intelligence and Statistics (AISTATS), pp. 249-256, 2010.
- Krizhevsky, A., Sutskever, I., and Hinton, G. ImageNet classification with deep convolutional neural networks. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems (NIPS), volume 25, pp. 1097-1105. 2012.
- LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Touretzky, D. S. (ed.), Advances in Neural Information Processing Systems (NIPS), volume 2, pp. 598-605.
- Morgan Kaufmann, 1990.
- Legg, S. and Hutter, M. Universal intelligence: a definition of machine intelligence. Minds and Machines, 17(4): 391-444, 2007.
- Lopez-Paz, D. and Ranzato, M. A. Gradient episodic mem- ory for continuum learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems (NIPS), volume 30, pp. 6449-6458.
- Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological Review, 97:285-308, 1990.
- Rebuffi, S., Kolesnikov, A., Sperl, G., and Lampert, C. iCaRL: incremental classifier and representation learn- ing. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
- Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. ArXiv, 1606.04671, 2016. · arXiv
- Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The German traffic sign recognition benchmark: a multi-class classification competition. In Proc. of the Int. Joint Conf. on Neural Networks (IJCNN), pp. 1453-1460, 2011.
- Venkatesan, R., Venkateswara, H., Panchanathan, S., and Li, B. A strategy for an uncompromising incremental learner. ArXiv, 1705.00744, 2017. · arXiv
Cited by
Sorted by citations
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
A Continual Learning Survey: Defying Forgetting in Classification Tasks
Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3057446 · Cited by 1,647
Towards Open World Object Detection
Joseph, K. J., Khan, S. H., Khan, F. S., & Balasubramanian, V. N. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00577 · Cited by 606
Metadata
- Authors
- Serrà, J., Surís, D., Miron, M., & Karatzoglou, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- arXiv
- 1801.01423
- OpenAlex
- W2782276075
- Semantic Scholar
- 3aa673abd49f0837ed2fbf5cffcada9ba6f48693
- Cited by
- 213
- References
- 34
- Citations by year
- 2025: 20 · 2024: 30 · 2023: 45 · 2022: 22 · 2021: 40 · 2020: 27 · 2019: 21 · 2018: 8
- FWCI
- 9.76
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- RECERCAT (Consorci de Serveis Universitaris de Catalunya) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Keywords
- catastrophic forgetting, hard attention, continual learning, sequential learning, attention masks, knowledge preservation, online learning, network compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
