PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
Open accessAbstract
This paper presents a method for adding multiple tasks to a single deep neural network while avoiding catastrophic forgetting. Inspired by network pruning techniques, we exploit redundancies in large deep networks to free up parameters that can then be employed to learn new tasks. By performing iterative pruning and network re-training, we are able to sequentially "pack" multiple tasks into a single network while ensuring minimal drop in performance and minimal storage overhead. Unlike prior work that uses proxy losses to maintain accuracy on older tasks, we always optimize for the task at hand. We perform extensive experiments on a variety of network architectures and large-scale datasets, and observe much better robustness against catastrophic forgetting than prior work. In particular, we are able to add three fine-grained classification tasks to a single ImageNet-trained VGG-16 network and achieve accuracies close to those of separately trained networks for each task.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
EWC, Progressive Neural Networks, and PackNet prevent catastrophic forgetting.
“EWC [9], Progressive Neural Networks [24], and PackNet [25] prevent catastrophic forgetting.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [25]
To avert catastrophic forgetting, various algorithms employ task-specific components or design separate networks for each task.
“To avert catastrophic forgetting, various algorithms[11, 48, 49] employ task-specific components or design separate networks for each task.”
From Few-Shot Class-Incremental Learning for Classification and Object Detection: A Survey (2025) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · see citing works
Incremental Learning of Object Detectors without Catastrophic Forgetting
Shmelkov, K., Schmid, C., & Alahari, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.368 · Cited by 559 · see citing works
Encoder Based Lifelong Learning
Rannen, A., Aljundi, R., Blaschko, M. B., & Tuytelaars, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.148 · Cited by 324 · see citing works
Flagged first, then by citations
Research works printed but not linked: 19
- R. Aljundi, P. Chakravarty, and T. Tuytelaars. Expert Gate: Lifelong learning with a network of experts. In CVPR, 2017. 1
- C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra. PathNet: Evo- lution channels gradient descent in super neural networks. arXiv:1701.08734, 2017. 2, 3 · arXiv
- R. M. French. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4):128-135, 1999.
- S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In ICML, 2015. 2
- S. Han, H. Mao, and W. J. Dally. Deep compression: Com- pressing deep neural networks with pruning, trained quanti- zation and huffman coding. In ICLR, 2016. 2
- S. Han, J. Pool, S. Narang, H. Mao, E. Gong, S. Tang, E. Elsen, P. Vajda, M. Paluri, J. Tran, et al. DSD: Dense- sparse-dense training for deep neural networks. In ICLR, 2017. 1, 2, 3, 7
- S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015. 1, 2, 3, 7
- G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS Workshop, 2014. 2
- G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 1, 3, 5
- I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks. In NIPS, 2016. 2, 8
- S.-W. Lee, J.-H. Kim, J.-W. Ha, and B.-T. Zhang. Overcom- ing catastrophic forgetting by incremental moment match- ing. In NIPS, 2017. 2, 3
- H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In ICLR, 2017. 1, 2, 8
- Z. Li and D. Hoiem. Learning without forgetting. In ECCV, 2016. 1, 2, 3, 4
- Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017. 1, 2, 8
- P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017. 1, 2, 8
- M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor- net: Imagenet classification using binary convolutional neu- ral networks. In ECCV, 2016. 2, 8
- A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv:1606.04671, 2016. 2, 3 · arXiv
- K. Simonyan and A. Zisserman. Very deep convolu- tional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 1, 3 · arXiv
- B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. TPAMI, 2017. 1, 3, 4, 5
Cited by
Sorted by citations
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547 · see citing works
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113 · see citing works
IL2M: Class Incremental Learning With Dual Memory
Belouadah, E., & Popescu, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00067 · Cited by 302 · see citing works
Overcoming catastrophic forgetting with hard attention to the task
Serrà, J., Surís, D., Miron, M., & Karatzoglou, A. (2018) · arXiv · doi:10.48550/arxiv.1801.01423 · Cited by 213 · see citing works
Topology-Preserving Class-Incremental Learning
Tao, X., Chang, X., Hong, X., Wei, X., & Gong, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_16 · Cited by 162 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122 · see citing works
Dynamic Memory to Alleviate Catastrophic Forgetting in Continual Learning with Medical Imaging
Perkonigg, M., Hofmanninger, J., Herold, C. J., Brink, J. A., Pianykh, O., Prosch, H., & Langs, G. (2021) · Nature Communications · doi:10.1038/s41467-021-25858-z · Cited by 106 · see citing works
Few-Shot Class-Incremental Learning for Classification and Object Detection: A Survey
Zhang, J., Liu, L., Silvén, O., Pietikäinen, M., & Hu, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3529038 · Cited by 84 · see citing works
“To avert catastrophic forgetting, various algorithms [11], [48], [49] employ task-specific components or design separate networks for each task.”
A survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65 · see citing works
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54 · see citing works
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47 · see citing works
DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues
Pan, K., Yin, Y., Wei, Y., Lin, F., Ba, Z., Liu, Z., Wang, Z., Cavallaro, L., & Ren, K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3612377 · Cited by 37 · see citing works
Data Augmented Flatness-aware Gradient Projection for Continual Learning
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., & Wang, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00518 · Cited by 18 · see citing works
Learning With Sharing: An Edge-Optimized Incremental Learning Method for Deep Neural Networks
Hussain, M. A., Huang, S.-A., & Tsai, T.-H. (2023) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2022.3210905 · Cited by 11 · see citing works
Revisiting Flatness-Aware Optimization in Continual Learning With Orthogonal Gradient Projection
Yang, E., Shen, L., Wang, Z., Liu, S., Guo, G., Wang, X., & Tao, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3539019 · Cited by 5 · see citing works
Online continual streaming learning for embedded space applications
Mazouz, A. E., & Nguyen, V.-T. (2024) · Journal of Real-Time Image Processing · doi:10.1007/s11554-024-01438-4 · Cited by 3 · see citing works
When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents
Hu, Q., Long, Q., & Wang, W. (2026) · arXiv · doi:10.48550/arxiv.2604.27003
“…data through regularization, replay, and architectural isolation or expansion (Kirkpatrick et al., 2017; Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Mallya and Lazebnik, 2018), with recent surveys extending this perspective to foundation models (Wang et al., 2024a; Wu et al., 2024).”
UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
Gu, X., Huang, Z., Gu, J.-C., & Zhang, K. (2025) · arXiv · doi:10.48550/arxiv.2505.14679
Kummara, H. (2026) · Cambridge Open Engage (preprint) · doi:10.33774/coe-2026-xpkvb
Huang, H., Ye, X., Mustafa, M. B., Dong, Q., Li, Y., Asemi, A., & Asemi, A. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02086-2
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
Belouadah, E., Popescu, A., & Kanellos, I. (2020) · arXiv · doi:10.48550/arxiv.2011.01844 · Cited by 235 · see citing works
“[30] is based on a pruning technique that iden-tifies redundant parameters and uses them to train the network on new tasks.”
Method
- Authors
- Mallya, A., & Lazebnik, S.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- ORCID
- Mallya, Arun
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 7765-7773
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2770882886
- Cited by
- 1,258
- References
- 9
- Citations by year
- 2026: 122 · 2025: 202 · 2024: 239 · 2023: 235 · 2022: 160 · 2021: 155 · 2020: 84 · 2019: 41 · 2018: 20
- FWCI
- 30.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1718221, 1563727)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- catastrophic forgetting, iterative pruning, multi-task learning, parameter-efficient fine-tuning, network redundancy, parameter reuse, storage efficiency, fine-grained classification, VGG16, continual learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
