Popular Ensemble Methods: An Empirical Study
Open accessAbstract
An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund & Shapire, 1996; Shapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier -- especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble's performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CONCERN EWS comprises a predictive model that uses an ensemble machine learning approach to process nursing surveillance patterns in the EHR, predicts clinical deterioration within the subsequent 24-hour period, and displays the risk score as a green, yellow or red icon on the EHR patient list.
“CONCERN EWS comprises a predictive model that uses an ensemble22 machine learning approach to process nursing surveillance patterns in the EHR, predicts clinical deterioration within the subsequent 24-hour period and displays the CONCERN EWS risk score as a green, yellow or red icon (non-interruptive alert) on the EHR patient list (main landing screen upon login) (Fig. 1).”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [22]
The models achieved more than 97% average accuracy and 95% average area under the curve.
“with more than 97% average accuracy and 95% average area under the curve (average performance reported for our ensemble models22) (Extended Data Table 5 and Supplementary Methods 2)”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [22]
The Results (p. 9) report test-set error rates for ensemble methods, and Section 3.5 (p. 13) discusses average error across 23 data sets. Neither passage reports more than 97% average accuracy or 95% average area under the curve; searches for those figures and for AUC found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredEnsembling multiple CNN models is a common deep learning approach for improving prediction results.
“Ensembling multiple CNN models is a common deep learning approach for improving prediction results [27,28,29].”
From Fully Automated Placental Volume Quantification From<scp>3D</scp>Ultrasound for Prediction of Small‐for‐Gestational‐Age Infants (2022) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
(1992) · Cited by 23,431 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Experiments with a new boosting algorithm
Freund, Y., & Schapire, R. E. (1996) · Cited by 8,189 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884 · see citing works
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works
The Strength of Weak Learnability
Schapire, R. E. (1990) · Machine Learning · doi:10.1007/bf00116037 · Cited by 4,600 · see citing works
Hansen, L. K., & Salamon, P. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.58871 · Cited by 4,260 · see citing works
The Handbook of brain theory and neural networks
(1996) · Choice Reviews Online · doi:10.5860/choice.33-4484 · Cited by 3,913 · see citing works
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · see citing works
An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants
Bauer, E. J., & Kohavi, R. (1999) · Machine Learning · doi:10.1023/a:1007515423169 · Cited by 2,627 · see citing works
Boosting the margin: a new explanation for the effectiveness of voting methods
Bartlett, P. L., Freund, Y., Lee, W. S., & Schapire, R. E. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691352 · Cited by 2,541 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1023/a:1018046112532 · Cited by 1,964 · see citing works
Neural Network Ensembles, Cross Validation, and Active Learning
Krogh, A., & Vedelsby, J. (1994) · Cited by 1,838 · see citing works
Combining forecasts: A review and annotated bibliography
(1990) · Long Range Planning · doi:10.1016/0024-6301(90)90316-v · Cited by 1,508 · see citing works
Quinlan, J. R. (1996) · National Conference on Artificial Intelligence · Cited by 1,251 · see citing works
On Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality
Friedman, J. H. (1997) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009778005914 · Cited by 1,038 · see citing works
Boosting the margin: A new explanation for the effectiveness of voting methods
Schapire, R. E., Freund, Y., Barlett, P., & Lee, W. S. (1997) · Cited by 711 · see citing works
Bias plus variance decomposition for zero-one loss functions
Kohavi, R., & Wolpert, D. H. (1996) · Cited by 626 · see citing works
Optimal Linear Combinations of Neural Networks
Hashem, S. (1997) · Neural Networks · doi:10.1016/s0893-6080(96)00098-6 · Cited by 499 · see citing works
Error-Correcting Output Coding Corrects Bias and Variance
Kong, E. B., & Dietterich, T. G. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50046-3 · Cited by 361 · see citing works
Learning with Ensembles: How over-fitting can be useful
Peter Sollich, A., & Krogh, A. (1995) · Cited by 353 · see citing works
Actively Searching for an Effective Neural Network Ensemble
Opitz, D. W., & SHAVLIK, J. W. (1996) · Connection Science · doi:10.1080/095400996116802 · Cited by 328 · see citing works
Self-citationError Reduction through Learning Multiple Descriptions
Ali, K., & Pazzani, M. J. (1994) · Machine Learning · doi:10.1023/a:1018249309965 · Cited by 296 · see citing works
Generating Accurate and Diverse Members of a Neural-Network Ensemble
Opitz, D. W., & SHAVLIK, J. W. (1995) · Cited by 279 · see citing works
Self-citationAn empirical evaluation of bagging and boosting
Maclin, R., & Opitz, D. W. (1997) · Cited by 265 · see citing works
Self-citationThe strength of weak learnability
Schapire, R. E. (1989) · Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.1989.63451 · Cited by 264 · see citing works
Drucker, H., & Cortes, C. (1995) · neural information processing systems · Cited by 233 · see citing works
Boosting in the limit: maximizing the margin of learned ensembles
Grove, A. J., & Schuurmans, D. (1998) · Cited by 232 · see citing works
Hybrid system for protein secondary structure prediction
Zhang, X., Mesirov, J. P., & Waltz, D. L. (1992) · Journal of Molecular Biology · doi:10.1016/0022-2836(92)90104-r · Cited by 228 · see citing works
Perrone, M. (1993) · Cited by 161 · see citing works
An empirical comparison of ID3 and back-propagation
Fisher, D., & McKusick, K. B. (1989) · Cited by 126 · see citing works
An experimental comparison of symbolic and connectionist learning algorithms
Mooney, R. J., SHAVLIK, J. W., Towell, G. G., et al. (1989) · Cited by 116 · see citing works
Combining Estimators Using Non-Constant Weighting Functions
Tresp, V., & Taniguchi, M. (1994) · Cited by 110 · see citing works
Maclin, R., & SHAVLIK, J. W. (1995) · Cited by 109 · see citing works
Self-citationImproving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks
Baxt, W. G. (1992) · Neural Computation · doi:10.1162/neco.1992.4.5.772 · Cited by 100 · see citing works
Multiple networks for function learning
Alpaydın, E. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1993.298539 · Cited by 94 · see citing works
Boosting and Other Machine Learning Algorithms
Drucker, H., Cortes, C., Jackel, L. D., LeCun, Y., & Vapnik, V. N. (1994) · Elsevier eBooks · doi:10.1016/b978-1-55860-335-6.50015-5 · Cited by 37 · see citing works
Lowering Variance of Decisions by Using Artificial Neural Network Portfolios
Mani, G. (1991) · Neural Computation · doi:10.1162/neco.1991.3.4.484 · Cited by 36 · see citing works
Ensembles as a sequence of classifiers
Asker, L., & Maclin, R. (1997) · Cited by 27 · see citing works
Self-citationBoosting classifiers regionally
Maclin, R. (1998) · Cited by 23 · see citing works
Self-citationFilter Selection Model for Generating Visual Motion Signals
Nowlan, S. J., & Sejnowski, T. J. (1992) · Cited by 21 · see citing works
A soft-competitive splitting rule for adaptive tree-structured neural networks
Perrone, M. (2003) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.227094 · Cited by 18 · see citing works
Feature Engineering and Classifier Selection: A Case Study in Venusian Volcano Detection
Asker, L., & Maclin, R. (1997) · Cited by 10 · see citing works
Self-citationA Soft-Competitive Splitting Rule for Adaptive Tree-Structured Neural Networks
Perrone, M. (1993) · doi:10.21236/ada264936 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Sagi, O., & Rokach, L. (2018) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1249 · Cited by 3,103 · see citing works
Ensembling neural networks: Many could be better than all
Zhou, Z.-H., Wu, J., & Tang, W. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00190-x · Cited by 1,964 · see citing works
Data-driven Soft Sensors in the process industry
Kadlec, P., Gabrys, B., & Strandt, S. (2009) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2008.12.012 · Cited by 1,845 · see citing works
Machine learning: a review of classification and combining techniques
Kotsiantis, S. B., Zaharakis, I. D., & Pintelas, P. E. (2006) · Artificial Intelligence Review · doi:10.1007/s10462-007-9052-3 · Cited by 1,582 · see citing works
Machine learning in manufacturing: Advantages, challenges, and applications
Wuest, T., Weimer, D., Irgens, C., & Thoben, K.-D. (2016) · Production & Manufacturing Research · doi:10.1080/21693277.2016.1192517 · Cited by 1,319 · see citing works
Methods of integrating data to uncover genotype-phenotype interactions
Ritchie, M. D., Holzinger, E. R., Li, R., Pendergrass, S. A., & Kim, D. (2015) · Nature Reviews Genetics · doi:10.1038/nrg3868 · Cited by 1,085 · see citing works
Recent advances in deep learning for object detection
Wu, X., Sahoo, D., & Hoi, S. C. H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.085 · Cited by 1,047 · see citing works
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810 · see citing works
Hate speech detection: Challenges and solutions
MacAvaney, S., Yao, H.-R., Yang, E., Russell, K., Goharian, N., & Frieder, O. (2019) · PLOS ONE · doi:10.1371/journal.pone.0221152 · Cited by 566 · see citing works
Ensemble methods for wind and solar power forecasting—A state-of-the-art review
Ren, Y., Suganthan, P. N., & Srikanth, N. (2015) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2015.04.081 · Cited by 435 · see citing works
XNOR-SRAM: In-Memory Computing SRAM Macro for Binary/Ternary Deep Neural Networks
Yin, S., Jiang, Z., Seo, J.-S., & Seok, M. (2020) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2019.2963616 · Cited by 429 · see citing works
Pre-trained convolutional neural networks as feature extractors for tuberculosis detection
Lopes, U. K., & Valiati, J. F. (2017) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2017.08.001 · Cited by 294 · see citing works
Human Activity Recognition via Hybrid Deep Learning Based Model
Khan, I. U., Afzal, S., & Lee, J. W. (2022) · Sensors · doi:10.3390/s22010323 · Cited by 256 · see citing works
Detection of anaemia from retinal fundus images via deep learning
Mitani, A., Huang, A., Venugopalan, S., Corrado, G. S., Peng, L., Webster, D. R., Hammel, N., Liu, Y., & Varadarajan, A. V. (2019) · Nature Biomedical Engineering · doi:10.1038/s41551-019-0487-z · Cited by 230 · see citing works
Deep Learning and Glaucoma Specialists
Phene, S., Dunn, R. C., Hammel, N., Liu, Y., Krause, J., Kitade, N., Schaekermann, M., Sayres, R., Wu, D. J., Bora, A., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2019.07.024 · Cited by 186 · see citing works
Fröhlich, F., Kessler, T., Weindl, D., Shadrin, A., Schmiester, L., Hache, H., Muradyan, A., Schütte, M., Lim, J.-H., Heinig, M., et al. (2018) · Cell Systems · doi:10.1016/j.cels.2018.10.013 · Cited by 143 · see citing works
Herzog, S. (2006) · New Directions for Institutional Research · doi:10.1002/ir.185 · Cited by 138 · see citing works
Health Big Data Analytics: A Technology Survey
Harerimana, G., Jang, B., Kim, J. W., & Park, H. K. (2018) · IEEE Access · doi:10.1109/access.2018.2878254 · Cited by 119 · see citing works
On combining classifiers using sum and product rules
Alexandre, L. A., Campilho, A. C., & Kamel, M. (2001) · Pattern Recognition Letters · doi:10.1016/s0167-8655(01)00073-3 · Cited by 117 · see citing works
Guber, A. K., Pachepsky, Y. A., van Genuchten, M. T., Rawls, W. J., Simunek, J., Jacques, D., Nicholson, T. J., & Cady, R. E. (2006) · Vadose Zone Journal · doi:10.2136/vzj2005.0111 · Cited by 91 · see citing works
Adepeju, M., Rosser, G., & Cheng, T. (2016) · International Journal of Geographical Information Science · doi:10.1080/13658816.2016.1159684 · Cited by 75 · see citing works
Rossetti, S. C., Dykes, P. C., Knaplund, C., Cho, S., Withall, J., Lowenthal, G., Albers, D., Lee, R. Y., Jia, H., Bakken, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03609-7 · Cited by 54 · see citing works
“Average performance reported for ensemble models 22 .”
InfluentialNasir, V., Kooshkbaghi, M., Cool, J., & Sassani, F. (2021) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-020-06473-6 · Cited by 48 · see citing works
Compression ensembles quantify aesthetic complexity and the evolution of visual art
Karjus, A., Canet Solà, M., Ohm, T., Ahnert, S. E., & Schich, M. (2023) · EPJ Data Science · doi:10.1140/epjds/s13688-023-00397-3 · Cited by 32 · see citing works
Sun, K., Roy, A., & Tobin, J. M. (2024) · Journal of Critical Care · doi:10.1016/j.jcrc.2024.154792 · Cited by 28 · see citing works
Rao, W., Qian, X., Fan, Y., & Liu, T. (2023) · Science of The Total Environment · doi:10.1016/j.scitotenv.2023.161543 · Cited by 15 · see citing works
Schwartz, N., Oguz, I., Wang, J., Pouch, A., Yushkevich, N., Parameshwaran, S., Gee, J., Yushkevich, P., & Oguz, B. (2022) · Journal of Ultrasound in Medicine · doi:10.1002/jum.15835 · Cited by 11 · see citing works
- Authors
- Opitz, D., & Maclin, R.
- Affiliations
- University of Montana (US)
- University of Minnesota, Duluth (US)
- ORCID
- Maclin, R.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 1999-08-01
- Volume
- 11
- Pages
- 169-198
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2100805904
- Cited by
- 2,953
- References
- 49
- Citations by year
- 2026: 89 · 2025: 148 · 2024: 176 · 2023: 225 · 2022: 198 · 2021: 253 · 2020: 224 · 2019: 201 · 2018: 131 · 2017: 128 · 2016: 107 · 2015: 126 · 2014: 98 · 2013: 99 · 2012: 110
- FWCI
- 19.45
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- ensemble methods, bagging, boosting, artificial neural networks, decision tree, classification accuracy, overfitting, noisy data, ensemble size, classifier combination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
