On the effect of calibration in classifier combination
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
A general approach to classifier combination considers each model as a probabilistic classifier which outputs a class membership posterior probability. In this general scenario, it is not only the quality and diversity of the models which are relevant, but the level of calibration of their estimated probabilities as well. In this paper, we study the role of calibration before and after classifier combination, focusing on evaluation measures such as MSE and AUC, which better account for good probability estimation than other evaluation measures. We present a series of findings that allow us to recommend several layouts for the use of calibration in classifier combination. We also empirically analyse a new non-monotonic calibration method that obtains better results for classifier combination than other monotonic calibration methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deviations from perfect calibration are very common in practice and may vary widely depending on the binary classification model that is used [20].”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [20]
In the citing paper
“Moreover, our experimental evaluation on a broad range of real datasets showed that ENIR outperforms IsoRegC and BBQ (i.e. a state-of-the-art binary classifier calibration method [20]).”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [20]
Citations
Cites
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,582
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,464
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
Combining Pattern Classifiers: Methods and Algorithms
Kuncheva, L. I. (2004) · Wiley eBooks · doi:10.1002/0471660264 · Cited by 3,005
Dietterich, T. G. (2000) · Machine Learning · doi:10.1023/a:1007607513941 · Cited by 2,941
Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy
Kuncheva, L. I., & Whitaker, C. J. (2003) · Machine Learning · doi:10.1023/a:1022859003006 · Cited by 2,335 · cited 1×
“On the other hand, it has been shown that an appropriate combination of several models can give better results than any of the single models alone [11][31][34][35][42], especially if the base classifiers are diverse [33].”
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276 · cited 1×
“In this paper, we will use Hand & Till’s extension [27], which is based on an aggregation of each class against each other, by using a uniform class distribution.”
In Defense of One-Vs-All Classification
Rifkin, R. M., & Klautau, A. (2004) · Journal of Machine Learning Research · Cited by 2,056
Using Bayesian model averaging to calibrate forecast ensembles.
Raftery, A. E., Gneiting, T., Balabdaoui, F., & Polakowski, M. (2005) · Monthly Weather Review · doi:10.1175/mwr2906.1 · Cited by 1,915 · cited 1×
“For example, in [38], the ExpectationMaximization algorithm was used to modify the weights of a Bayesian Model Averaging [28] method and to obtain a calibrated ensemble, but the effect of calibration methods before the combination was not studied.”
Advances in Large-Margin Classifiers
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1113.001.0001 · Cited by 1,869
Order Restricted Statistical Inference.
Santner, T., Robertson, T., Wright, F. T., & Dykstra, R. L. (1990) · Journal of the American Statistical Association · doi:10.2307/2289813 · Cited by 1,586
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · cited 2×
“Binning averaging was proposed by [47] as a method for binary classifiers where a (validation) dataset is split into”
Regression Analysis when the Dependent Variable Is Truncated Normal
Amemiya, T. (1973) · Econometrica · doi:10.2307/1914031 · Cited by 1,129
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
An Empirical Distribution Function for Sampling with Incomplete Information
Ayer, M., Brunk, H. D., Ewing, G. M., Reid, W. T., & Silverman, E. (1955) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728423 · Cited by 904
An experimental comparison of performance measures for classification
Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.010 · Cited by 889
Self-citationA theoretical study on six classifier fusion strategies
Kuncheva, L. I. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982906 · Cited by 720
Data mining in metric space: an empirical analysis of supervised learning performance criteria
Caruana, R., & Niculescu-Mizil, A. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014063 · Cited by 303
Sutton, C., Rohanimanesh, K., & McCallum, A. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015422 · Cited by 284
Gama, J., & Brazdil, P. (2000) · Machine Learning · doi:10.1023/a:1007652114878 · Cited by 281
Review of Classifier Combination Methods
Tulyakov, S., Jaeger, S., Govindaraju, V., & Doermann, D. (2008) · Studies in Computational Intelligence · doi:10.1007/978-3-540-76280-5_14 · Cited by 219
Order Restricted Statistical Inference.
Hayter, T., Robertson, T., Wright, F. T., & Dykstra, R. L. (1990) · The Statistician · doi:10.2307/2348204 · Cited by 205
Diversity in multiple classifier systems
Kuncheva, L. I. (2005) · Information Fusion · doi:10.1016/j.inffus.2004.04.009 · Cited by 165
Volume under the ROC Surface for Multi-class Problems
Ferri, C., Hernández-Orallo, J., & Salido, M. A. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-39857-8_12 · Cited by 162
Self-citationMeasuring, refining and calibrating speaker and language information extracted from speech
Brümmer, N. (2010) · SUNScholar (Stellenbosch University) · Cited by 148
Getting the Most Out of Ensemble Selection
Caruana, R., Munson, A., & Niculescu-Mizil, A. (2006) · Sixth International Conference on Data Mining (ICDM'06) · doi:10.1109/icdm.2006.76 · Cited by 142
Properties and Benefits of Calibrated Classifiers
Cohen, I., & Goldszmidt, M. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30116-5_14 · Cited by 125
Scalar and Vector Partitions of the Probability Score: Part I. Two-State Situation
Murphy, A. H. (1972) · Journal of Applied Meteorology · doi:10.1175/1520-0450(1972)011<0273:savpot>2.0.co;2 · Cited by 122
Ferri, C., Flach, P. A., & Hernández‐Orallo, J. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015395 · Cited by 91
Self-citationData mining and decision support : integration and collaboration
Mladenić, D. (2003) · Medical Entomology and Zoology · Cited by 90
Intelligent Data Engineering and Automated Learning – IDEAL 2018
(2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03496-2 · Cited by 87
Khor, K., Ting, C., & Phon-Amnuaisuk, S. (2010) · Applied Intelligence · doi:10.1007/s10489-010-0263-y · Cited by 80
Calibration of Machine Learning Models
Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2010) · Handbook of Research on Machine Learning Applications and Trends · doi:10.4018/978-1-60566-766-9.ch006 · Cited by 59
Self-citationThe Combination of Text Classifiers Using Reliability Indicators
Bennett, P. N., Dumais, S., & Horvitz, E. (2004) · Information Retrieval · doi:10.1023/b:inrt.0000048491.59134.94 · Cited by 57
Decision Support for Data Mining
Flach, P. A., Blockeel, H., Ferri, C., Hernández‐Orallo, J., & Struyf, J. (2003) · Springer eBooks · doi:10.1007/978-1-4615-0286-9_7 · Cited by 39
Self-citationHybrid ensemble approach for classification
Verma, B., & Hassan, S. Z. (2009) · Applied Intelligence · doi:10.1007/s10489-009-0194-7 · Cited by 35
A new selective neural network ensemble with negative correlation
Lee, H., Kim, E., & Witold Pedrycz, C. (2012) · Applied Intelligence · doi:10.1007/s10489-012-0342-3 · Cited by 32
Calibration of Machine Learning Models
Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2012) · Machine Learning · doi:10.4018/978-1-60960-818-7.ch104 · Cited by 27
Self-citationClassification rules in standardized partition spaces
Garczarek, U. (2002) · Technische Universität Dortmund Eldorado (Technische Universität Dortmund) · doi:10.17877/de290r-14912 · Cited by 27
EEM: evolutionary ensembles model for activity recognition in Smart Homes
Fahim, M., Fatima, I., Lee, S., & Lee, Y.-K. (2012) · Applied Intelligence · doi:10.1007/s10489-012-0359-7 · Cited by 27
Multivariate calibration of classifier scores into the probability space
Gebel, M. (2009) · Technische Universität Dortmund Eldorado (Technische Universität Dortmund) · doi:10.17877/de290r-863 · Cited by 25
Proceedings of the 13th European Conference on Machine Learning
Elomaa, T., Mannila, H., & Toivonen, H. T. T. (2002) · Cited by 21
Random projections for linear SVM ensembles
Maudes, J., Rodríguez, J. J., García‐Osorio, C., & Pardo, C. (2011) · Applied Intelligence · doi:10.1007/s10489-011-0283-2 · Cited by 18
A low variance error boosting algorithm
Wang, C., & Hunter, A. (2009) · Applied Intelligence · doi:10.1007/s10489-009-0172-0 · Cited by 12
Ensembles of ARTMAP-based neural networks: an experimental study
Canuto, A. M. P., Santos, A. D. M., & Vargas, R. R. D. (2009) · Applied Intelligence · doi:10.1007/s10489-009-0199-2 · Cited by 12
Applying correlation to enhance boosting technique using genetic programming as base learner
de Souza, L. V., Pozo, A., da Rosa, J. C., & Neto, A. C. (2009) · Applied Intelligence · doi:10.1007/s10489-009-0166-y · Cited by 12
Building reliable metaclassifiers for text learning
Carbonell, J., Lafferty, J., & Bennett, P. N. (2006) · Cited by 11
Similarity-Binning Averaging: A Generalisation of Binning Calibration
Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04394-9_42 · Cited by 9
Self-citationVERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
(1996)
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 2×
“Of the last family of measures, Mean Squared Error (MSE) or, for two classes, Brier Score [9] penalises strong deviations from the true probability:”
Cited by
Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500
Krauss, C., Do, X. A., & Huck, N. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2016.10.031 · Cited by 750
Musolf, A. M., Holzinger, E. R., Malley, J. D., & Wilson, J. (2021) · Human Genetics · doi:10.1007/s00439-021-02402-z · Cited by 168
Development of genome-derived tumor type prediction to inform clinical cancer care
Penson, A., Camacho, N., Zheng, Y., Varghese, A. M., Al-Ahmadie, H., Razavi, P., Chandarlapaty, S., Vallejo, C. E., Vakiani, E., Gilewski, T., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2019.3985 · Cited by 115
Evidential calibration of binary SVM classifiers
Xu, P., Davoine, F., Zha, H., & Denœux, T. (2016) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2015.05.002 · Cited by 83
Sieberts, S. K., Zhu, F., García-García, J., Stahl, E. A., Pratap, A., Pandey, G. S., Pappas, D. A., Aguilar, D., Anton, B., Bonet, J., et al. (2016) · Nature Communications · doi:10.1038/ncomms12460 · Cited by 93
Metadata
- Authors
- Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J.
- Affiliations
- Universitat Politècnica de València (ES)
- Corresponding author
- Bella, Antonio
- ORCID
- Ferri, Cèsar
- Venue
- Applied Intelligence
- Published
- 2012-09-28
- Volume
- 38
- Issue
- 4
- Pages
- 566-585
- ISSN
- 0924-669X (ISSN-L)
- 1573-7497
- OpenAlex
- W1974191159
- Cited by
- 45
- References
- 56
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 2 · 2022: 1 · 2021: 5 · 2020: 5 · 2019: 3 · 2018: 2 · 2017: 4 · 2016: 6 · 2015: 4 · 2014: 7 · 2013: 2 · 2012: 1
- FWCI
- 4.41
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- www.springer.com/tdm (tdm)
- Locations
- Applied Intelligence (publishedVersion)
- RiuNet (Universitat Politècnica de València) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Keywords
- classifier combination, probability calibration, ensemble learning, posterior probability estimation, model diversity, evaluation metrics, mean squared error, AUC
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.