The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
Open accessAbstract
In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to the state-of-the-art in lifelong learning without forgetting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Tensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Caltech-UCSD Birds-200-2011 Dataset
Wah, C. (2024) · CaltechAUTHORS (California Institute of Technology) · doi:10.57702/i9oeewf6 · Cited by 3,525
Natural Gradient Works Efficiently in Learning
Amari, S.-I. (1998) · Neural Computation · doi:10.1162/089976698300017746 · Cited by 2,681
Continual Learning Through Synaptic Intelligence
Zenke, F., Poole, B., & Ganguli, S. (2017) · arXiv · doi:10.48550/arxiv.1703.04200 · Cited by 1,784
Gradient Episodic Memory for Continual Learning
Lopez-Paz, D., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1706.08840 · Cited by 1,403
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_33 · Cited by 1,091
Li, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659
Expert Gate: Lifelong Learning with a Network of Experts
Aljundi, R., Chakravarty, P., & Tuytelaars, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.753 · Cited by 619
Human action recognition by learning bases of action attributes and parts
Yao, B., Jiang, X., Khosla, A., Lin, A. L., Guibas, L., & Fei-Fei, L. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126386 · Cited by 609
Restructuring of deep neural network acoustic models with singular value decomposition
Xue, J., Li, J., & Gong, Y. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-552 · Cited by 407
Revisiting Natural Gradient for Deep NetworksPreprint
Pascanu, R., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3584 · Cited by 355
Encoder Based Lifelong Learning
Rannen, A., Aljundi, R., Blaschko, M. B., & Tuytelaars, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.148 · Cited by 324
Overcoming Catastrophic Forgetting by Incremental Moment Matching
Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., & Zhang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08475 · Cited by 255
Overcoming catastrophic forgetting with hard attention to the task
Serrà, J., Surís, D., Miron, M., & Karatzoglou, A. (2018) · arXiv · doi:10.48550/arxiv.1801.01423 · Cited by 213
Revisiting Natural Gradient for Deep NetworksPreprint
Pascanu, R., & Bengio, Y. (2014) · arXiv (Cornell University) · Cited by 170
Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F. J., & Schmidhuber, J. (2013) · neural information processing systems · Cited by 151
Natural Neural NetworksPreprint
Desjardins, G., Simonyan, K., Pascanu, R., & Kavukcuoglu, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.00210 · Cited by 147
Note on the quadratic penalties in elastic weight consolidation
Huszár, F. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717042115 · Cited by 134
The Task Rehearsal Method of Life-Long Learning: Overcoming Impoverished Data
Silver, D. L., & Mercer, R. E. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47922-8_8 · Cited by 127
Domain-Adaptive Deep Network Compression
Masana, M., van de Weijer, J., Herranz, L., Bagdanov, A. D., & Álvarez, J. M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.460 · Cited by 56
Self-citation
Cited by
A Continual Learning Survey: Defying Forgetting in Classification Tasks
Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3057446 · Cited by 1,647
Self-citationA continual learning survey: Defying forgetting in classification tasks
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., & Tuytelaars, T. (2019) · arXiv · doi:10.48550/arxiv.1909.08383 · Cited by 1,647
Self-citationA Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., & van de Weijer, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3213473 · Cited by 740
Self-citationFew-Shot Class-Incremental Learning
Tao, X., Hong, X., Chang, X., Dong, S., Wei, X., & Gong, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01220 · Cited by 499
Metadata
- Authors
- Liu, X., Masana, M., Herranz, L., Van de Weijer, J., Lopez, A. M., & Bagdanov, A. D.
- Venue
- 2018 24th International Conference on Pattern Recognition (ICPR)
- Publisher
- IEEE
- Published
- 2018-08
- Pages
- 2262-2268
- ISSN
- 1041-3278 (ISSN-L)
- OpenAlex
- W2963788399
- Cited by
- 259
- References
- 26
- FWCI
- 10.40
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.