Learn++: an incremental learning algorithm for supervised neural networks
Abstract
We introduce Learn++, an algorithm for incremental training of neural network (NN) pattern classifiers. The proposed algorithm enables supervised NN paradigms, such as the multilayer perceptron (MLP), to accommodate new data, including examples that correspond to previously unseen classes. Furthermore, the algorithm does not require access to previously used data during subsequent incremental learning sessions, yet at the same time, it does not forget previously acquired knowledge. Learn++ utilizes ensemble of classifiers by generating multiple hypotheses using training data sampled according to carefully tailored distributions. The outputs of the resulting classifiers are combined using a weighted majority voting procedure. We present simulation results on several benchmark datasets as well as a real-world classification task. Initial results indicate that the proposed algorithm works rather well in practice. A theoretical upper bound on the error of the classifiers constructed by Learn++ is also provided.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Learn++ has the ability to learn new classes, a small number of parameters, and short training time, but suffers from data imbalance when learning new classes.
“As an algorithm proposed earlier, the ability to learn new classes is one of the main features and advantages of learn++. It also has the advantages of a small number of parameters and short training time. However, it suffers from data imbalance when learning new classes.”
From An Appraisal of Incremental Learning Methods (2020) · cites this paper as [24]
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466 · see citing works
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,491 · see citing works
Experiments with a new boosting algorithm
Freund, Y., & Schapire, R. E. (1996) · Cited by 8,189 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,312 · see citing works
The Strength of Weak Learnability
Schapire, R. E. (1990) · Machine Learning · doi:10.1007/bf00116037 · Cited by 4,600 · see citing works
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,730 · see citing works
Boosting the margin: a new explanation for the effectiveness of voting methods
Bartlett, P. L., Freund, Y., Lee, W. S., & Schapire, R. E. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691352 · Cited by 2,541 · see citing works
The Weighted Majority Algorithm
Littlestone, N., & Warmuth, M. K. (1994) · Information and Computation · doi:10.1006/inco.1994.1009 · Cited by 2,001 · see citing works
Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., & Rosen, D. B. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.159059 · Cited by 1,977 · see citing works
Nonlinear neural networks: Principles, mechanisms, and architectures
Grossberg, S. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90021-4 · Cited by 1,623 · see citing works
Carpenter, G. A., Grossberg, S., & Reynolds, J. H. (1991) · Neural Networks · doi:10.1016/0893-6080(91)90012-t · Cited by 981 · see citing works
Gaussian ARTMAP: A Neural Network for Fast Incremental Learning of Noisy Multidimensional Maps
Williamson, J. R. (1996) · Neural Networks · doi:10.1016/0893-6080(95)00115-8 · Cited by 312 · see citing works
Error Reduction through Learning Multiple Descriptions
Ali, K., & Pazzani, M. J. (1994) · Machine Learning · doi:10.1023/a:1018249309965 · Cited by 296 · see citing works
Handbook of Biosensors and Electronic Noses: Medicine, Food, and the Environment
Kress-Rogers, E. (1996) · Medical Entomology and Zoology · Cited by 258 · see citing works
Constructive neural-network learning algorithms for pattern classification
Parekh, R. G., Yang, J., & Honavar, V. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.839013 · Cited by 221 · see citing works
Self-citationCombinations of weak classifiers
Ji, C., & Ma, S. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.554189 · Cited by 159 · see citing works
Incremental backpropagation learning networks
Fu, L., Hsu, H.-H., & Principe, J. C. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.501732 · Cited by 103 · see citing works
Life-long learning Cell Structures—continuously learning without catastrophic interference
Hamker, F. H. (2001) · Neural Networks · doi:10.1016/s0893-6080(01)00018-1 · Cited by 102 · see citing works
An Incremental Adaptive Network for On-line Supervised Learning and Probability Estimation
Lim, C. P., & Harrison, R. F. (1997) · Neural Networks · doi:10.1016/s0893-6080(96)00123-2 · Cited by 87 · see citing works
Incremental learning methods with retrieving of interfered patterns
Yamauchi, K., Yamaguchi, N., & Ishii, N. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.809080 · Cited by 87 · see citing works
An effective neuro-fuzzy paradigm for machinery condition health monitoring
Yen, G. G., & Meesad, P. (2001) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/3477.938258 · Cited by 67 · see citing works
Ellipsoid ART and ARTMAP for incremental clustering and classification
Anagnostopoulos, G. C., & Georgiopoulos, M. (2002) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2001.939535 · Cited by 59 · see citing works
Convergent on-line algorithms for supervised learning in neural networks
Grippo, L. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.883426 · Cited by 45 · see citing works
Performance and efficiency: recent advances in supervised learning
Ma, S., & Ji, C. (1999) · Proceedings of the IEEE · doi:10.1109/5.784228 · Cited by 44 · see citing works
An incremental learning algorithm that optimizes network size and sample size in one trial
Zhang, B. (1994) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1994.374165 · Cited by 39 · see citing works
INSS: A hybrid system for constructive machine learning
Osório, F. S., & Amy, B. (1999) · Neurocomputing · doi:10.1016/s0925-2312(98)00124-6 · Cited by 37 · see citing works
RKHS-based functional analysis for exact incremental learning
Vijayakumar, S., & Ogawa, H. (1999) · Neurocomputing · doi:10.1016/s0925-2312(99)00112-5 · Cited by 32 · see citing works
Incremental knowledge acquisition in supervised learning networks
Fu, L. (1996) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.541338 · Cited by 29 · see citing works
Rademacher penalization applied to fuzzy ARTMAP and boosted ARTMAP
Verzi, S., Heileman, G. L., Georgiopoulus, M., & Healy, M. J. R. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2001.939530 · Cited by 27 · see citing works
A clustering approach to incremental learning for feedforward neural networks
Engelbrecht, A. P., & Brits, R. (2002) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2001.938474 · Cited by 27 · see citing works
A smart algorithm for incremental learning
Wang, E. H.-C., & Kuh, A. (2003) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.227182 · Cited by 21 · see citing works
An heuristic pattern correction scheme for GRNNs and its application to speech recognition
Hoya, T., & Constantinides, A. G. (2002) · doi:10.1109/nnsp.1998.710665 · Cited by 18 · see citing works
Incremental learning using the time delay neural network
Vo, M. T. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1994.389577 · Cited by 16 · see citing works
Incremental learning with rule-based neural networks
Higgins, C. M., & Goodman, R. M. (2002) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.155294 · Cited by 16 · see citing works
Rangarajan, S., Jalote, P., & Tripathi, S. K. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.238570 · Cited by 14 · see citing works
On-line learning using hierarchical mixtures of experts
Tham, C. (1995) · doi:10.1049/cp:19950580 · Cited by 4 · see citing works
A training approach based on linear separability analysis for layered perceptrons
Zhang, D. D., Kamel, M. S., & Elmasry, M. I. (1994) · doi:10.1109/icnn.1994.374217 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988 · see citing works
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · see citing works
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270 · see citing works
Ensemble learning for data stream analysis: A survey
Krawczyk, B., Minku, L. L., Gama, J., Stefanowski, J., & Woźniak, M. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.004 · Cited by 1,050 · see citing works
A Survey on Ensemble Learning for Data Stream Classification
Gomes, H. M., Barddal, J. P., Enembreck, F., & Bifet, A. (2018) · ACM Computing Surveys · doi:10.1145/3054925 · Cited by 579 · see citing works
Measuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672 · see citing works
Incremental Learning of Object Detectors without Catastrophic Forgetting
Shmelkov, K., Schmid, C., & Alahari, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.368 · Cited by 559 · see citing works
Incremental on-line learning: A review and comparison of state of the art algorithms
Losing, V., Hammer, B., & Wersing, H. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.06.084 · Cited by 382 · see citing works
An Appraisal of Incremental Learning Methods
Luo, Y., Yin, L., Bai, W., & Mao, K. (2020) · Entropy · doi:10.3390/e22111190 · Cited by 120 · see citing works
Federated Learning Empowered Real-Time Medical Data Processing Method for Smart Healthcare
Guo, K., Chen, T., Ren, S., Li, N., Hu, M., & Kang, J. (2024) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3185395 · Cited by 63 · see citing works
Wang, Z., Li, H.-X., & Chen, C. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2927320 · Cited by 39 · see citing works
Incremental supervised learning: Algorithms and applications in pattern recognition
Chefrour, A. (2019) · Evolutionary Intelligence · doi:10.1007/s12065-019-00203-y · Cited by 42 · see citing works
A Strategy for an Uncompromising Incremental Learner
Venkatesan, R., Venkateswara, H., Panchanathan, S., & Li, B. (2017) · arXiv · doi:10.48550/arxiv.1705.00744 · Cited by 37 · see citing works
Incremental learning of concept drift in Multiple Instance Learning for industrial visual inspection
Mera, C., Orozco-Alzate, M., & Branch, J. (2019) · Computers in Industry · doi:10.1016/j.compind.2019.04.006 · Cited by 18 · see citing works
- Authors
- Polikar, R., Upda, L., Upda, S. S., & Honavar, V.
- Affiliations
- Iowa State University (US)
- Michigan State University (US)
- ORCID
- Honavar, V.
- Venue
- IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews)
- Published
- 2001
- Volume
- 31
- Issue
- 4
- Pages
- 497-508
- ISSN
- 1094-6977 (ISSN-L)
- OpenAlex
- W2103753221
- Cited by
- 888
- References
- 41
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 29 · 2023: 44 · 2022: 57 · 2021: 65 · 2020: 73 · 2019: 76 · 2018: 44 · 2017: 44 · 2016: 32 · 2015: 40 · 2014: 43 · 2013: 29 · 2012: 32
- FWCI
- 8.15
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Keywords
- incremental learning, neural network classifier, multilayer perceptron, ensemble classifiers, weighted majority voting, unseen classes, catastrophic forgetting, training data sampling, error bounds
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review