The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Ensemble based systems in decision making
Abstract
In matters of great importance that have financial, medical, social, or other implications, we often seek a second opinion before making a decision, sometimes a third, and sometimes many more. In doing so, we weigh the individual opinions, and combine them through some thought process to reach a final decision that is presumably the most informed one. The process of consulting "several experts" before making a final decision is perhaps second nature to us; yet, the extensive benefits of such a process in automated decision making applications have only recently been discovered by computational intelligence community. Also known under various other names, such as multiple classifier systems, committee of classifiers, or mixture of experts, ensemble based systems have shown to produce favorable results compared to those of single-expert systems for a broad range of applications and under a variety of scenarios. Design, implementation and application of such systems are the main topics of this article. Specifically, this paper reviews conditions under which ensemble based systems may be more beneficial than their single classifier counterparts, algorithms for generating individual components of the ensemble systems, and various procedures through which the individual classifiers can be combined. We discuss popular ensemble based algorithms, such as bagging, boosting, AdaBoost, stacked generalization, and hierarchical mixture of experts; as well as commonly used combination rules, including algebraic combination of outputs, voting based techniques, behavior knowledge space, and decision templates. Finally, we look at current and future research directions for novel applications of ensemble systems. Such applications include incremental learning, data fusion, feature selection, learning with missing features, confidence estimation, and error correcting output codes; all areas in which ensemble systems have shown great promise
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
A Mathematical Theory of Evidence
Shafer, G. (1976) · Princeton University Press eBooks · doi:10.1515/9780691214696 · Cited by 9,083
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
Lecture Notes in Artificial Intelligence
Brézillon, P., & Bouquet, P. (1999) · Cited by 7,473
The random subspace method for constructing decision forests
Ho, T. K. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.709601 · Cited by 6,842
Upper and Lower Probabilities Induced by a Multivalued Mapping
Dempster, A. P. (1967) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177698950 · Cited by 6,126
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Upper and Lower Probabilities Induced by a Multivalued Mapping
Dempster, A. P. (2008) · Studies in Fuzziness and Soft Computing · doi:10.1007/978-3-540-44792-4_3 · Cited by 5,296
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,290
The Strength of Weak Learnability
Schapire, R. E. (1990) · Machine Learning · doi:10.1007/bf00116037 · Cited by 4,600
Hansen, L. K., & Salamon, P. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.58871 · Cited by 4,260
Abe, S. (2001) · Springer eBooks · doi:10.1007/978-1-4471-0285-4 · Cited by 3,721
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
Dietterich, T. G. (2000) · Machine Learning · doi:10.1023/a:1007607513941 · Cited by 2,941
Solving Multiclass Learning Problems via Error-Correcting Output Codes
Dietterich, T. G., & Bakiri, G. (1995) · Journal of Artificial Intelligence Research · doi:10.1613/jair.105 · Cited by 2,739
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants
Bauer, E., & Kohavi, R. (1999) · Machine Learning · doi:10.1023/a:1007515423169 · Cited by 2,638
Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
Kuncheva, L. I., & Whitaker, C. J. (2003) · Machine Learning · doi:10.1023/a:1022859003006 · Cited by 2,335
Catastrophic forgetting in connectionist networks
French, R. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01294-2 · Cited by 2,311
Boosting the margin: a new explanation for the effectiveness of voting methods
Bartlett, P., Freund, Y., Lee, W. S., & Schapire, R. E. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691352 · Cited by 2,306
Methods of combining multiple classifiers and their applications to handwriting recognition
Xu, L., Krzyzak, A., & Suen, C. Y. (1992) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.155943 · Cited by 2,179
Monti, S., Tamayo, P., Mesirov, J., & Golub, T. (2003) · Machine Learning · doi:10.1023/a:1023949509487 · Cited by 2,116
The Weighted Majority Algorithm
Littlestone, N., & Warmuth, M. K. (1994) · Information and Computation · doi:10.1006/inco.1994.1009 · Cited by 1,975
Quinlan, J. R. (1996) · National Conference on Artificial Intelligence · Cited by 1,586
Decision combination in multiple classifier systems
Ho, T. K., Hull, J. J., & Srihari, S. N. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.273716 · Cited by 1,511
Arcing classifier (with discussion and a rejoinder by the author)
Breiman, L. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691079 · Cited by 1,105
Decision templates for multiple classifier fusion: an experimental comparison
Kuncheva, L. I., Bezdek, J. C., & Duin, R. P. W. (2001) · Pattern Recognition · doi:10.1016/s0031-3203(99)00223-x · Cited by 1,003
Combination of multiple classifiers using local accuracy estimates
Woods, K., Kegelmeyer, W. P., & Bowyer, K. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.588027 · Cited by 952
Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers
Allwein, E. L., Schapire, R. E., & Singer, Y. (2000) · Cited by 914
Learn++: an incremental learning algorithm for supervised neural networks
Polikar, R., Upda, L., Upda, S. S., & Honavar, V. (2001) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/5326.983933 · Cited by 888
Self-citationA theoretical study on six classifier fusion strategies
Kuncheva, L. I. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982906 · Cited by 720
Information combination operators for data fusion: a comparative review with classification
Bloch, I. (1996) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.477860 · Cited by 639
A method of combining multiple experts for the recognition of unconstrained handwritten numerals
Huang, Y. S., & Suen, C. Y. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.368145 · Cited by 564
A neural network classifier based on Dempster-Shafer theory
Denoeux, T. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.833094 · Cited by 432
Combining the results of several neural network classifiers
Rogova, G. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90099-x · Cited by 402
Combining multiple classifiers by averaging or by multiplying?
Tax, D. M. J., van Breukelen, M., Duin, R. P. W., & Kittler, J. (2000) · Pattern Recognition · doi:10.1016/s0031-3203(99)00138-7 · Cited by 377
Methods For Combining Experts' Probability Assessments
Jacobs, R. A. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.867 · Cited by 372
Boosting and Other Ensemble Methods
Drucker, H., Cortes, C., Jackel, L. D., LeCun, Y., & Vapnik, V. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1289 · Cited by 357
Consensus theoretic classification methods
Benediktsson, J. A., & Swain, P. H. (1992) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.156582 · Cited by 357
Switching between selection and fusion in combining classifiers: an experiment
Kuncheva, L. I. (2002) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/3477.990871 · Cited by 348
Combining multiple neural networks by fuzzy integral for robust classification
Cho, S.-B., & Kim, J. H. (1995) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.364825 · Cited by 340
A composite classifier system design: Concepts and methodology
Dasarathy, B. V., & Sheela, B. V. (1979) · Proceedings of the IEEE · doi:10.1109/proc.1979.11321 · Cited by 340
Classifier Ensembles for Changing Environments
Kuncheva, L. I. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-25966-4_1 · Cited by 335
Analysis of decision boundaries in linearly combined neural classifiers
Tumer, K., & Ghosh, J. (1996) · Pattern Recognition · doi:10.1016/0031-3203(95)00085-2 · Cited by 335
Pasting Small Votes for Classification in Large Databases and On-Line
Breiman, L. (1999) · Machine Learning · doi:10.1023/a:1007563306331 · Cited by 317
Democracy in neural nets: Voting schemes for classification
Battiti, R., & Colla, A. M. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90046-9 · Cited by 308
A theoretical and experimental analysis of linear combiners for multiple classifier systems
Fumera, G., & Roli, F. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.109 · Cited by 307
Optimal combinations of pattern classifiers
Lam, L., & Suen, C. Y. (1995) · Pattern Recognition Letters · doi:10.1016/0167-8655(95)00050-q · Cited by 306
Convergence results for the EM approach to mixtures of experts architectures
Jordan, M. I., & Xu, L. (1995) · Neural Networks · doi:10.1016/0893-6080(95)00014-3 · Cited by 305
Ensemble Learning Using Decorrelated Neural Networks
ROSEN, B. E. (1996) · Connection Science · doi:10.1080/095400996116820 · Cited by 248
Experiments with Classifier Combining Rules
Duin, R. P. W., & Tax, D. M. J. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_2 · Cited by 235
An approach to the automatic design of multiple classifier systems
Giacinto, G., & Roli, F. (2001) · Pattern Recognition Letters · doi:10.1016/s0167-8655(00)00096-9 · Cited by 197
Multiple network fusion using fuzzy logic
Cho, S.-B., & Kim, J. H. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.363487 · Cited by 197
Methods for Designing Multiple Classifier Systems
Roli, F., Giacinto, G., & Vernazza, G. L. (2001) · Lecture notes in computer science · doi:10.1007/3-540-48219-9_8 · Cited by 183
Diversity in Neural Network Ensembles
Gavin Brown, B. (2004) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 180
Ichihashi, H., SHIRAI, T., Nagasaka, K., et al. (1996) · Fuzzy Sets and Systems · doi:10.1016/0165-0114(95)00247-2 · Cited by 141
Multiclassifier Systems: Back to the Future
Ghosh, J. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45428-4_1 · Cited by 122
Experimental evaluation of expert fusion strategies
Alkoot, F. M., & Kittler, J. (1999) · Pattern Recognition Letters · doi:10.1016/s0167-8655(99)00107-5 · Cited by 118
Input Decimation Ensembles: Decorrelation through Dimensionality Reduction
Oza, N. C., & Tumer, K. (2001) · Lecture notes in computer science · doi:10.1007/3-540-48219-9_24 · Cited by 98
The behavior-knowledge space method for combination of multiple classifiers
Huang, Y., & Suen, C. Y. (1993) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1993.1626170 · Cited by 90
Carpenter, G. A., Martens, S., & Ogas, O. J. (2005) · Neural Networks · doi:10.1016/j.neunet.2004.12.003 · Cited by 83
Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions
French, R. M. (1999) · Cited by 73
Boosting with Averaged Weight Vectors
Oza, N. C. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44938-8_2 · Cited by 61
On fusers that perform better than best sensor
Rao, N. S. V. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.946993 · Cited by 60
Learn++.MT: A New Approach to Incremental Learning
Muhlbaier, M. D., Topalis, A., & Robi Polikar, A. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-25966-4_5 · Cited by 58
Long, P. M., & Vega, V. B. (2003) · Machine Learning · doi:10.1023/a:1023937123600 · Cited by 52
AveBoost2: Boosting for Noisy Data
Oza, N. C. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-25966-4_3 · Cited by 44
Robi Polikar, A., Удпа, Л., Udpa, S. S., & Honavar, V. (2004) · IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control · doi:10.1109/tuffc.2004.1324403 · Cited by 41
The pandemonium system of reflective agents
Śmieja, F. J. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.478395 · Cited by 40
Feature Subsets for Classifier Combination: An Enumerative Experiment
Kuncheva, L. I., & Whitaker, C. J. (2001) · Lecture notes in computer science · doi:10.1007/3-540-48219-9_23 · Cited by 36
On linear mixture of expert approaches to information retrieval
Fan, W., Gordon, M., & Pathak, P. (2004) · Decision Support Systems · doi:10.1016/j.dss.2004.11.014 · Cited by 29
Incremental Learning With Balanced Update on Receptive Fields for Multi-Sensor Data Fusion
Su, J., Wang, J., & Xi, Y. (2004) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2002.806485 · Cited by 21
Research works printed but not linked: 22
- 10.1109/IJCNN.2004.1380080 · DOI
- 10.1016/S0167-6393(03)00032-3 · doi:10.1016/s0167-6393(03)00032-3
- 10.1016/0165-0114(94)90023-X · doi:10.1016/0165-0114(94)90023-x
- grim, 2001, Information analysis of multiple classifier fusion, Proc 1st Int Workshop on Multiple Classifier Systems, 2096, 168, 10.1007/3-540-48219-9_17 · doi:10.1007/3-540-48219-9_17
- 10.1016/S0165-0114(99)00161-X · doi:10.1016/s0165-0114(99)00161-x
- 10.1007/BF00117809 · doi:10.1007/bf00117809
- 10.1007/11494683_33 · doi:10.1007/11494683_33
- 10.1109/72.501737 · doi:10.1109/72.501737
- chawla, 2002, Distributed pasting of small votes, Proc 1st Int Workshop on Multiple Classifier Systems, 2364, 52, 10.1007/3-540-45428-4_5 · doi:10.1007/3-540-45428-4_5
- ayad, 2005, Cluster-Based Cumulative Ensembles, 6th Int Workshop on Multiple Classifier Sys, 3541, 236, 10.1007/11494683_24 · doi:10.1007/11494683_24
- smith, 2005, Decoding Rules for Error Correcting Output Code Ensembles, 6th Int Workshop on Multiple Classifier Sys, 3541, 53, 10.1007/11494683_6 · doi:10.1007/11494683_6
- rajan, 2004, Lecture Notes in Computer Science, 3077, 283, 10.1007/978-3-540-25966-4_28 · DOI
- strehl, 2002, Cluster Ensembles-? Knowledge Reuse Framework for Combining Multiple Partitions, Journal of Machine Learning Research, 3, 583
- 10.1007/BF00118940 · doi:10.1007/bf00118940
- 10.2307/2348873 · doi:10.2307/2348873
- 10.1016/0167-8655(93)90046-G · doi:10.1016/0167-8655(93)90046-g
- 10.1109/ICSMC.2004.1399793 · doi:10.1109/icsmc.2004.1399793
- 10.1109/IGARSS.2001.976668 · doi:10.1109/igarss.2001.976668
- 10.1177/002029400103400802 · doi:10.1177/002029400103400802
- 10.1109/ICPR.2004.1334133 · doi:10.1109/icpr.2004.1334133
- 10.1109/IJCNN.2003.1223406 · doi:10.1109/ijcnn.2003.1223406
- authors, 2005, Proceedings of International Workshop on Multiple Classifier Systems (2000-2005)
Cited by
A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection
Buczak, A. L., & Guven, E. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2015.2494502 · Cited by 3,332
Sagi, O., & Rokach, L. (2018) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1249 · Cited by 3,103
Link prediction in complex networks: A survey
Lü, L., & Zhou, T. (2011) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2010.11.027 · Cited by 2,819
Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., & Herrera, F. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2161285 · Cited by 2,794
Rokach, L. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-009-9124-7 · Cited by 2,657
Metadata
- Authors
- Polikar, R.
- Venue
- IEEE Circuits and Systems Magazine
- Published
- 2006
- Volume
- 6
- Issue
- 3
- Pages
- 21-45
- ISSN
- 0163-6812 (ISSN-L)
- 1531-636X
- OpenAlex
- W2167917621
- Cited by
- 2,938
- References
- 76
- FWCI
- 60.09
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.