Boosting Neural Networks
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Boosting is a general method for improving the performance of learning algorithms. A recently proposed boosting algorithm, Ada Boost, has been applied with great success to several benchmark machine learning problems using mainly decision trees as base classifiers. In this article we investigate whether Ada Boost also works as well with neural networks, and we discuss the advantages and drawbacks of different versions of the Ada Boost algorithm. In particular, we compare training methods based on sampling the training set and weighting the cost function. The results suggest that random resampling of the training data is not the main explanation of the success of the improvements brought by Ada Boost. This is in contrast to bagging, which directly aims at reducing variance and for which random resampling is essential to obtain the reduction in generalization error. Our system achieves about 1.4% error on a data set of on-line handwritten digits from more than 200 writers. A boosted multilayer network achieved 1.5% error on the UCI letters and 8.1% error on the UCI satellite data set, which is significantly better than boosted decision trees.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884 · cited 1×
“Other attemps to understand boosting theoretically can be found in Schapire et al., 1997; Breiman 1997a, 1998; Friedman, Hastie, and Tibshirani, 1998; and Schapire, 1999.”
BackgroundThe Strength of Weak Learnability
Schapire, R. E. (1990) · Machine Learning · doi:10.1007/bf00116037 · Cited by 4,600
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · cited 1×
“Formulated in the context of the bias-variance decomposition ( Geman, Bienenstok, & Doursat, 1992 ), bagging improves generalization performance due to a reduction in variance while maintaining or only slightly increasing bias.”
MethodDietterich, T. G. (2000) · Machine Learning · doi:10.1023/a:1007607513941 · Cited by 2,941
An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants
Bauer, E., & Kohavi, R. (1999) · Machine Learning · doi:10.1023/a:1007515423169 · Cited by 2,638
Improved boosting algorithms using confidence-rated predictions
Schapire, R. E., & Singer, Y. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279960 · Cited by 2,547
Neural Network Ensembles, Cross Validation, and Active Learning
Krogh, A., & Vedelsby, J. (1994) · Cited by 1,838
Boosting a Weak Learning Algorithm by Majority
Freund, Y. (1995) · Information and Computation · doi:10.1006/inco.1995.1136 · Cited by 1,674
Quinlan, J. R. (1996) · National Conference on Artificial Intelligence · Cited by 1,586
Arcing classifier (with discussion and a rejoinder by the author)
Breiman, L. (1998) · The Annals of Statistics · doi:10.1214/aos/1024691079 · Cited by 1,105
Boosting the margin: A new explanation for the effectiveness of voting methods
Schapire, R. E., Freund, Y., Barlett, P., & Lee, W. S. (1997) · Cited by 711
Bias plus variance decomposition for zero-one loss functions
Kohavi, R., & Wolpert, D. H. (1996) · Cited by 652
Adaptive Game Playing Using Multiplicative Weights
Freund, Y., & Schapire, R. E. (1999) · Games and Economic Behavior · doi:10.1006/game.1999.0738 · Cited by 515
Game theory, on-line prediction and boosting
Freund, Y., & Schapire, R. E. (1996) · Proceedings of the ninth annual conference on Computational learning theory - COLT '96 · doi:10.1145/238061.238163 · Cited by 383
Error-Correcting Output Coding Corrects Bias and Variance
Kong, E. B., & Dietterich, T. G. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50046-3 · Cited by 365
An empirical evaluation of bagging and boosting
Maclin, R., & Opitz, D. W. (1997) · Cited by 265
Boosting in the limit: maximizing the margin of learned ensembles
Grove, A. J., & Schuurmans, D. (1998) · Cited by 231
Drucker, H., & Cortes, C. (1995) · neural information processing systems · Cited by 229
Perrone, M. (1993) · Cited by 159
Schapire, R. E. (1999) · Lecture notes in computer science · doi:10.1007/3-540-49097-3_1 · Cited by 122
Efficient Training of Feed-Forward Neural Networks
Møller, M. F. (1993) · DAIMI Report Series · doi:10.7146/dpb.v22i464.6937 · Cited by 62
(1996)
Cited by
Cost-sensitive boosting for classification of imbalanced data
Sun, Y., Kamel, M. S., Wong, A. K. C., & Wang, Y. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2007.04.009 · Cited by 1,423
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810
“Ensemble learning ap-3 Observe that this strategy holds even when the entire dataset D is replaced by the dataset for a particular country D c . proaches utilize boosting algorithms [46], even for ensembles of neural networks [47], to enhance accuracy.”
BackgroundDownscaling of precipitation for climate change scenarios: A support vector machine approach
Tripathi, S., Srinivas, V. V., & Nanjundiah, R. S. (2006) · Journal of Hydrology · doi:10.1016/j.jhydrol.2006.04.030 · Cited by 607
Bagging, Boosting and the Random Subspace Method for Linear Classifiers
Skurichina, M., & Duin, R. P. W. (2002) · Pattern Analysis & Applications · doi:10.1007/s100440200011 · Cited by 521
Tardaguila, M., de la Fuente, L., Marti, C., Pereira, C., Pardo-Palacios, F. J., del Risco, H., Ferrell, M., Mellado, M., Macchietto, M., Verheggen, K., et al. (2018) · Genome Research · doi:10.1101/gr.222976.117 · Cited by 454
Metadata
- Authors
- Schwenk, H., & Bengio, Y.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2000-08-01
- Volume
- 12
- Issue
- 8
- Pages
- 1869-1887
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2164921999
- Cited by
- 297
- References
- 25
- FWCI
- 6.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.