Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification. However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing DCNN to learn new tasks while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned tasks. An updated network, for learning new set of classes, is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We employed a `clone-and-branch' technique with calibration, which allows the network to learn new tasks (containing classes with similar features as old tasks) one after another without any performance loss in old tasks. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing), while achieving energy efficiency, reduction in storage requirements, memory access and training time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 15×
“We could share lot more of the learning parameters from the base network in ResNet34 compared to ResNet18.”
MethodInfluentialOvercoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215 · cited 1×
“Note that we do not consider the energy expended in off-chip data movement (movement of weight from DRAM to local memory in cores) and inter-core data movement (over network) as these can vary based on the layer configurations, chip size, network design and several optimizations obtained from the software layers [33].”
BackgroundLi, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · cited 3×
“Even with periodical utilization of old data samples, [25] reaches 62% (top 1) accuracy.”
BackgroundLearning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849 · cited 2×
“Many of them focus on learning new tasks from fewer samples [6], [12] utilizing transfer learning techniques.”
BackgroundLi, Z., & Hoiem, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_37 · Cited by 659
Expert Gate: Lifelong Learning with a Network of Experts
Aljundi, R., Chakravarty, P., & Tuytelaars, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.753 · Cited by 619 · cited 1×
“[1] proposed a gating approach to select the model that can provide the best performance for the current task.”
BackgroundIncremental Learning of Object Detectors without Catastrophic Forgetting
Shmelkov, K., Schmid, C., & Alahari, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.368 · Cited by 559
PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference
Ankit, A., Hajj, I. E., Chalamalasetti, S. R., Ndu, G., Foltin, M., Williams, R. S., Faraboschi, P., Hwu, W.-M. W., Strachan, J. P., Roy, K., et al. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304049 · Cited by 424
Self-citationTree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning
Roy, D., Panda, P., & Roy, K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.010 · Cited by 257 · cited 1×
“[27] where the network grows in a tree-like manner to accommodate the new classes.”
BackgroundSelf-citationFALCON: Feature driven selective classification for energy-efficient image recognition
Panda, P., Ankit, A., Wijesinghe, P., & Roy, K. (2017) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2017.2681075 · Cited by 21 · cited 1×
“The multi-stage network will allow selective activation of a task specific branch [20] while other branches will be inactive.”
Background
Cited by
Recent advances in deep learning
Wang, X., Zhao, Y., & Pourpanah, F. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01096-5 · Cited by 321
Tree-CNN: A hierarchical Deep Convolutional Neural Network for incremental learning
Roy, D., Panda, P., & Roy, K. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.09.010 · Cited by 257
Self-citationIncremental Learning Techniques for Semantic Segmentation
Michieli, U., & Zanuttigh, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00400 · Cited by 256
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120
“Sarwar et al. [35] designed a deep convolutional neural network model that incrementally grows up with new tasks and the basic backbone was retained and shared for previous tasks.”
BackgroundIncremental supervised learning: Algorithms and applications in pattern recognition
Chefrour, A. (2019) · Evolutionary Intelligence · doi:10.1007/s12065-019-00203-y · Cited by 42
“Sarwar et al. [33] develop a Deep convolutional neural network (DCNN).”
Background
Metadata
- Authors
- Sarwar, S. S., Ankit, A., & Roy, K.
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 4615-4628
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2773246426
- Cited by
- 120
- References
- 15
- FWCI
- 8.52
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.