Cited by 1,590Domain Adaptation and Few-Shot LearningNeural Networks and ApplicationsAdvanced Neural Network Applications

The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index; none of the 28 sources its claims cite is checked for support yet.

Memory Aware Synapses: Learning What (not) to Forget

Open access

Abstract

Humans can learn in a continuous manner. Old rarely utilized knowledge can be overwritten by new incoming information while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning so far has focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. In this paper, we argue that, given the limited model capacity and the unlimited new information to be learned, knowledge has to be preserved or erased selectively. Inspired by neuroplasticity, we propose a novel approach for lifelong learning, coined Memory Aware Synapses (MAS). It computes the importance of the parameters of a neural network in an unsupervised and online manner. Given a new sample which is fed to the network, MAS accumulates an importance measure for each parameter of the network, based on how sensitive the predicted output function is to a change in this parameter. When learning a new task, changes to important parameters can then be penalized, effectively preventing important knowledge related to previous tasks from being overwritten. Further, we show an interesting connection between a local version of our method and Hebb's rule,which is a model for the learning process in the brain. We test our method on a sequence of object recognition tasks and on the challenging problem of learning an embedding for predicting $ $ triplets. We show state-of-the-art performance and, for the first time, the ability to adapt the importance of the parameters based on unlabeled data towards what the network needs (not) to forget, which may vary depending on test conditions.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

Memory Aware Synapses (MAS) estimates each network parameter’s importance from the sensitivity of the learned function’s output, not the loss, so importance can be computed online and unsupervised on labeled or unlabeled data. It penalizes changes to important parameters when learning new tasks and reports state-of-the-art lifelong learning results on object recognition and fact-learning embedding tasks.

Key findings

  • MAS reduces catastrophic forgetting in object recognition: in two-task sequences it achieves average forgetting around 1%, while FineTune drops 8–13%; in an 8-task sequence MAS has average forgetting of 0.49% and average accuracy 52.695, compared with SI at 50.49 and Finetune at 32.67.
  • In fact learning on the 6DS dataset, MAS reaches overall MAP 0.29 on a 4-task random split versus SI 0.25 and Finetune 0.18; using both training and test data for importance further improves MAS to 0.30.
  • MAS can adapt importance to unlabeled test conditions: for the Krishna Kamal flower subset, forgetting was 2% versus 8% on the whole Flower dataset, and last-task accuracy was 84% instead of 69% without adaptation; on a sports-fact subset, MAS reached 0.50 MAP while Finetune and SI were close to 0.20.
  • A local variant, l-MAS, is linked to Hebbian learning and computes importance from neuron activation correlations; it is faster but less accurate, with average forgetting of 3% versus 1% for global MAS.

Methods

  • MAS is a model-based lifelong learning method: after convergence, it accumulates an importance weight Ω_ij for each parameter by measuring how sensitive the squared ℓ2 norm of the learned function output is to parameter changes over available data points.
  • When learning a new task, it adds a regularizer that penalizes changes to parameters with high Ω_ij, allowing low-importance parameters to adapt; Ω can be updated online and computed on training, validation, or unlabeled test data.
  • Experiments use AlexNet pretrained on ImageNet for sequences of object recognition datasets (MIT Scenes, Caltech-UCSD Birds, Oxford Flowers, Stanford Cars, FGVCAircraft, VOC Actions, Letters, SVHN) and a VGG-16-based embedding model on the 6DS fact-learning dataset, reporting classification accuracy and mean average precision (MAP).

Limitations

  • The paper states that l-MAS is faster but less accurate than global MAS, with average forgetting of 3% versus 1%.
  • The paper states that λ was set to 1 in object recognition experiments without tuning, because it assumes no access to previous task data.
  • Not explicitly stated as a limitation, but evident from the text: the object recognition setup assumes an oracle at test time to select the task-specific classification head, and separate unshared heads per task.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-030-01219-9_9

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

34 Own interpretation · 28 Own result · 15 Not checked yet · 12 No basis given · 1 Own method

Verify all 28 references — free account

Show all 37 claims

  • It may state two separate claims as one. 39
  • Probably not a substantial claim. 19
53 of 53
  • The authors state this as an assumption.

    No access to previous task data is available.

    “note that no tuning of λ was performed as we assume no access to previous task data.”
  • The authors state this as an assumption.

    In the standard LLL setup, when training a task, only the data related to that task is accessible.

    “The assumption of this setup is that, when training a task, only the data related to that task is accessible.”
  • The authors state this as an assumption.

    In the object recognition experiments, different classification layers for each task remain unshared.

    “Note that this assumes having different classification layers for each task (different 'heads') that remain unshared.”
  • The paper relies on this without support.

    In the absence of error signals, synapses connecting biological neurons strengthen or weaken based on the concurrence of the connected neurons activations.

    “In the absence of error signals, synapses connecting biological neurons strengthen or weaken based on the concurrence of the connected neurons activations.”
  • The authors state this as an assumption.

    A good importance measure would give very low importance values to unused parameters and high values for those that are crucial for the task at hand.

    “Ideally, a good importance measure would give very low importance values to the unused parameters and high values for those that are crucial for the task at hand.”
  • The authors state this as an assumption.

    A plot where points are closely lying around a straight line indicates that the parameters from the two Ωs have similar importance values.

    “A plot where the points are closely lying around a straight line indicates that the parameters from the two Ω s have similar importance values.”
  • The authors state this as an assumption.

    A plot where points are spread further from such a line and scattered among the plotted area indicates a lower correlation between the Ωs.

    “A plot where the points are spread further from such a line and scattered among the plotted area indicates a lower correlation between the Ω s.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 37 references

Research works printed but not linked: 5

  1. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results (2012). http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html
  2. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012)
  3. Pentina, A., Lampert, C.H.: Lifelong learning with non-iid tasks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 1540–1548 (2015)
  4. Pentina, A., Lampert, C.H.: Lifelong learning with non-iid tasks. In: Advances in Neural Information Processing Systems, pp. 1540–1548 (2015)
  5. Quadrianto, N., Petterson, J., Smola, A.J.: Distribution matching for transduction. In: Advances in Neural Information Processing Systems, pp. 1500–1508 (2009)

Cited by

Sorted by citations

Items 801–900 of 1,000

Metadata

Authors
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., & Tuytelaars, T.
Affiliations
  • IMEC (BE)
  • Meta (United States) (US)
Corresponding author
Aljundi, Rahaf
Venue
Lecture Notes in Computer Science
Published
2018
Pages
144-161
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
OpenAlex
W2768412495
Cited by
1,590
References
37
Citations by year
2026: 166 · 2025: 281 · 2024: 288 · 2023: 295 · 2022: 184 · 2021: 203 · 2020: 105 · 2019: 52 · 2018: 15
FWCI
38.29
Type
conference paper
Language
en
Open access
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Lirias (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
Fonds Wetenschappelijk Onderzoek
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
catastrophic forgetting, lifelong learning, parameter importance, neuroplasticity, Hebbian learning, online learning, object recognition, knowledge preservation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review