Expert Gate: Lifelong Learning with a Network of Experts
Abstract
In this paper we introduce a model of lifelong learning, based on a Network of Experts. New tasks / experts are learned and added to the model sequentially, building on what was learned before. To ensure scalability of this process, data from previous tasks cannot be stored and hence is not available when learning a new task. A critical issue in such context, not addressed in the literature so far, relates to the decision which expert to deploy at test time. We introduce a set of gating autoencoders that learn a representation for the task at hand, and, at test time, automatically forward the test sample to the relevant expert. This also brings memory efficiency as only one expert network has to be loaded into memory at any given time. Further, the autoencoders inherently capture the relatedness of one task to another, based on which the most relevant prior model to be used for training a new expert, with fine-tuning or learning-without-forgetting, can be selected. We evaluate our method on image classification and video prediction problems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The standard LLL setup consists of a sequence of disjoint tasks learned one after the other, where tasks may correspond to different datasets or different splits of a dataset without overlap in category labels.
“It focuses on image classification and consists of a sequence of disjoint tasks , which are learned one after the other. Tasks may correspond to different datasets, or different splits of a dataset, without overlap in category labels.”
From Memory Aware Synapses: Learning What (not) to Forget (2018) · cites this paper as [1]
Data-based approaches use data from the new task to approximate the performance of the previous tasks, and this works best if the data distribution mismatch between tasks is limited.
“Data-based approaches [1,17,28,34] use data from the new task to approximate the performance of the previous tasks. This works best if the data distribution mismatch between tasks is limited.”
From Memory Aware Synapses: Learning What (not) to Forget (2018) · cites this paper as [1]
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
(2016) · Cited by 11,679 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,048 · see citing works
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,037 · see citing works
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,261 · see citing works
3D Object Representations for Fine-Grained Categorization
Krause, J., Stark, M. M., Deng, J., & Fei-Fei, L. (2013) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.77 · Cited by 3,523 · see citing works
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · see citing works
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · see citing works
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · see citing works
The PASCAL visual object classes challenge 2006 (VOC2006) results
Everingham, M., Zisserman, A., Williams, C. K. I., & Gool, L. V. (2006) · Cited by 2,079 · see citing works
Context-dependent computation by recurrent dynamics in prefrontal cortex
Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013) · Nature · doi:10.1038/nature12742 · Cited by 1,807 · see citing works
Quattoni, A., & Torralba, A. B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206537 · Cited by 1,533 · see citing works
Auto-association by multilayer perceptrons and singular value decomposition
Bourlard, H., & Kamp, Y. (1988) · Biological Cybernetics · doi:10.1007/bf00332918 · Cited by 1,332 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
Fine-Grained Visual Classification of AircraftPreprint
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., & Vedaldi, A. (2013) · HAL · doi:10.48550/arxiv.1306.5151 · Cited by 1,250 · see citing works
Caruana, R. (1998) · Cited by 1,110 · see citing works
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089 · see citing works
Jia, X., Brabandere, B. D., Tuytelaars, T., & Gool, L. V. (2016) · neural information processing systems · Cited by 681 · see citing works
Self-citationLi, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659 · see citing works
Caruana, R. (1998) · Learning to Learn · doi:10.1007/978-1-4615-5529-2_5 · Cited by 577 · see citing works
The Need for Biases in Learning Generalizations
Mitchell, T. M. (2007) · Cited by 520 · see citing works
Multi-task learning for classification with dirichlet process priors
Xue, Y., Liao, X., Carin, L., Krishnapuram, B., & Bartlett, P. (2007) · Cited by 478 · see citing works
Dynamic Filter NetworksPreprint
Brabandere, B. D., Jia, X., Tuytelaars, T., & Gool, L. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09673 · Cited by 461 · see citing works
Self-citationCHARACTER RECOGNITION IN NATURAL IMAGES
de Campos, T. E., Babu, B. R., & Varma, M. (2009) · Cited by 449 · see citing works
On rectified linear units for speech processing
Zeiler, M. D., Ranzato, M., Monga, R., Mao, M., Yang, K., Le, Q. V., Nguyen, P., Senior, A., Vanhoucke, V., Dean, J., et al. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638312 · Cited by 442 · see citing works
Clustered Multi-Task Learning: A Convex FormulationPreprint
Jacob, L., Bach, F., & Vert, J. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0809.2085 · Cited by 417 · see citing works
What Regularized Auto-Encoders Learn from the Data Generating DistributionPreprint
Alain, G., & Bengio, Y. (2013) · Cited by 319 · see citing works
(2013) · Cited by 290 · see citing works
CHARACTER RECOGNITION IN NATURAL IMAGES
de Campos, T. E., Babu, B. R., & Varma, M. (2009) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0001770102730280 · Cited by 276 · see citing works
Learning Task Grouping and Overlap in Multi-task LearningPreprint
Kumar, A., & Daumé, H. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6417 · Cited by 270 · see citing works
What regularized auto-encoders learn from the data-generating distribution
AlainGuillaume, & BengioYoshua (2014) · Journal of Machine Learning Research · doi:10.5555/2627435.2750359 · Cited by 166 · see citing works
The Task Rehearsal Method of Life-Long Learning: Overcoming Impoverished Data
Silver, D. L., & Mercer, R. E. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47922-8_8 · Cited by 127 · see citing works
A Unified Energy-Based Framework for Unsupervised Learning
Ranzato, M., Boureau, Y.-L., Chopra, S., et al. (2007) · International Conference on Artificial Intelligence and Statistics · Cited by 105 · see citing works
Network of Experts for Large-Scale Image Categorization
Ahmed, K., Baig, M. H., & Torresani, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_32 · Cited by 96 · see citing works
Clustering Learning Tasks and the Selective Cross-Task Transfer of Knowledge
Thrun, S., & O’Sullivan, J. A. (1998) · Springer eBooks · doi:10.1007/978-1-4615-5529-2_10 · Cited by 73 · see citing works
Knowledge Transfer in Deep Block-Modular Neural Networks
Terekhov, A. V., Montone, G., & O'Regan, J. K. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22979-9_27 · Cited by 70 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Memory Aware Synapses: Learning What (not) to Forget
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., & Tuytelaars, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_9 · Cited by 1,590 · see citing works
Self-citationA Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113 · see citing works
“Expert Gate [243] employs a mixture of experts to learn incremental tasks, expanding one expert as each task is introduced.”
BackgroundIL2M: Class Incremental Learning With Dual Memory
Belouadah, E., & Popescu, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00067 · Cited by 302 · see citing works
Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning
Roy, D., Panda, P., & Roy, K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.010 · Cited by 257 · see citing works
Encoder Based Lifelong Learning
Rannen, A., Aljundi, R., Blaschko, M. B., & Tuytelaars, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.148 · Cited by 324 · see citing works
Self-citationLifelong Learning via Progressive Distillation and Retrospection
Hou, S., Pan, X., Loy, C. C., Wang, Z., & Lin, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_27 · Cited by 190 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54 · see citing works
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47 · see citing works
Continual Learning for Generative Retrieval over Dynamic Corpora
Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., & Cheng, X. (2023) · Proceedings of the 32nd ACM International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614821 · Cited by 40 · see citing works
Adversarial Feature Alignment: Avoid Catastrophic Forgetting in Incremental Task Lifelong Learning
Yao, X., Huang, T., Wu, C., Zhang, R.-X., & Sun, L. (2019) · Neural Computation · doi:10.1162/neco_a_01232 · Cited by 27 · see citing works
Self-Net: Lifelong Learning via Continual Self-Modeling
Mandivarapu, J. K., Camp, B., & Estrada, R. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00019 · Cited by 14 · see citing works
Revisiting Flatness-Aware Optimization in Continual Learning With Orthogonal Gradient Projection
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., Wang, X., & Tao, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3539019 · Cited by 5 · see citing works
Online continual streaming learning for embedded space applications
Mazouz, A. E., & Nguyen, V.-T. (2024) · Journal of Real-Time Image Processing · doi:10.1007/s11554-024-01438-4 · Cited by 3 · see citing works
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Belouadah, E., Popescu, A., & Kanellos, I. (2020) · arXiv · doi:10.48550/arxiv.2011.01844 · Cited by 235 · see citing works
“C om pl ex ity The model evolves by adding parameters and weights to interconnect them [42, 48, 56] or small networks [2] to include new knowledge.”
BackgroundInfluential
- Authors
- Aljundi, R., Chakravarty, P., & Tuytelaars, T.
- Affiliations
- IMEC (BE)
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 7120-7129
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2554616628
- Cited by
- 619
- References
- 42
- Citations by year
- 2026: 47 · 2025: 81 · 2024: 108 · 2023: 117 · 2022: 84 · 2021: 73 · 2020: 56 · 2019: 35 · 2018: 17 · 2017: 1
- FWCI
- 13.60
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Lirias (KU Leuven) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- lifelong learning, memory efficiency, learning without forgetting, fine-tuning, task relatedness, image classification, video prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
