On the effect of calibration in classifier combination
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
A general approach to classifier combination considers each model as a probabilistic classifier which outputs a class membership posterior probability. In this general scenario, it is not only the quality and diversity of the models which are relevant, but the level of calibration of their estimated probabilities as well. In this paper, we study the role of calibration before and after classifier combination, focusing on evaluation measures such as MSE and AUC, which better account for good probability estimation than other evaluation measures. We present a series of findings that allow us to recommend several layouts for the use of calibration in classifier combination. We also empirically analyse a new non-monotonic calibration method that obtains better results for classifier combination than other monotonic calibration methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deviations from perfect calibration are very common in practice and may vary widely depending on the binary classification model that is used [20].”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [20]
In the citing paper
“Moreover, our experimental evaluation on a broad range of real datasets showed that ENIR outperforms IsoRegC and BBQ (i.e. a state-of-the-art binary classifier calibration method [20]).”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [20]
Citations
Cites
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,582
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,464
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118
Combining Pattern Classifiers: Methods and Algorithms
Bagui, S. C. (2005) · Technometrics · doi:10.1198/tech.2005.s320 · Cited by 3,202
Combining Pattern Classifiers: Methods and Algorithms
Kuncheva, L. I. (2004) · Wiley eBooks · doi:10.1002/0471660264 · Cited by 3,005
Dietterich, T. G. (2000) · Machine Learning · doi:10.1023/a:1007607513941 · Cited by 2,941
Cited by
Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500
Krauss, C., Do, X. A., & Huck, N. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2016.10.031 · Cited by 750
Musolf, A. M., Holzinger, E. R., Malley, J. D., & Wilson, J. (2021) · Human Genetics · doi:10.1007/s00439-021-02402-z · Cited by 168
Development of genome-derived tumor type prediction to inform clinical cancer care
Penson, A., Camacho, N., Zheng, Y., Varghese, A. M., Al-Ahmadie, H., Razavi, P., Chandarlapaty, S., Vallejo, C. E., Vakiani, E., Gilewski, T., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2019.3985 · Cited by 115
Evidential calibration of binary SVM classifiers
Xu, P., Davoine, F., Zha, H., & Denœux, T. (2016) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2015.05.002 · Cited by 83
Sieberts, S. K., Zhu, F., García-García, J., Stahl, E. A., Pratap, A., Pandey, G. S., Pappas, D. A., Aguilar, D., Anton, B., Bonet, J., et al. (2016) · Nature Communications · doi:10.1038/ncomms12460 · Cited by 93
Metadata
- Authors
- Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J.
- Affiliations
- Universitat Politècnica de València (ES)
- Corresponding author
- Bella, Antonio
- ORCID
- Ferri, Cèsar
- Venue
- Applied Intelligence
- Published
- 2012-09-28
- Volume
- 38
- Issue
- 4
- Pages
- 566-585
- ISSN
- 0924-669X (ISSN-L)
- 1573-7497
- OpenAlex
- W1974191159
- Cited by
- 45
- References
- 56
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 2 · 2022: 1 · 2021: 5 · 2020: 5 · 2019: 3 · 2018: 2 · 2017: 4 · 2016: 6 · 2015: 4 · 2014: 7 · 2013: 2 · 2012: 1
- FWCI
- 4.41
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- www.springer.com/tdm (tdm)
- Locations
- Applied Intelligence (publishedVersion)
- RiuNet (Universitat Politècnica de València) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Keywords
- classifier combination, probability calibration, ensemble learning, posterior probability estimation, model diversity, evaluation metrics, mean squared error, AUC
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the effect of calibration in classifier combination. https://app.cochat.ai/papers/10.1007/s10489-012-0388-2
- The paper
- Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2013). On the effect of calibration in classifier combination. Applied Intelligence. https://doi.org/10.1007/s10489-012-0388-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.