Binary Classifier Calibration Using a Bayesian Non-Parametric Approach
Open accessAbstract
Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in Data mining. This paper presents two new non-parametric methods for calibrating outputs of binary classification models: a method based on the Bayes optimal selection and a method based on the Bayesian model averaging. The advantage of these methods is that they are independent of the algorithm used to learn a predictive model, and they can be applied in a post-processing step, after the model is learned. This makes them applicable to a wide variety of machine learning models and methods. These calibration methods, as well as other methods, are tested on a variety of datasets in terms of both discrimination and calibration performance. The results show the methods either outperform or are comparable in performance to the state-of-the-art calibration methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
mPAVA finds the entire solution path for near isotonic regression in O(N log N) time and O(N) memory.
“mPAVA finds the whole solution path in O(N log N), and needs O(N) memory space.”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [25]
The Discussion (page unavailable) mentions the PAV algorithm and gives a running time for isotonic regression, but does not discuss mPAVA or a near-isotonic solution path. Searches for mPAVA, near-isotonic regression, solution-path complexity, and memory use found no passage supporting the claimed O(N log N) time or O(N) space bounds. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe near-isotonic regression problem yields a unique solution p-hat-lambda for a given lambda.
“It yields a unique solution p̂λ, where the use of the subscript λ emphasizes the dependency of the final solution to the value of λ.”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [25]
The Introduction and Discussion discuss isotonic regression and relaxing its monotonicity constraint, but do not address whether a near-isotonic regression problem has a unique solution. In Methods §2.1, λ is the mean parameter of a Poisson prior on bin boundaries, not a parameter of a near-isotonic solution. Searches for near-isotonic regression and a unique solution for a given λ found no supporting passage in our copy; page numbers were unavailable. (LLM, conf=0.76)
Experimental — accuracy not yet measuredAs λ grows, bins can only merge and cannot split, so the solution path is piecewise linear and its breakpoints occur when two bins merge.
“Consequently, the solution path remains piecewise linear as λ increases, and the breakpoints happen when two bins merge together.”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [25]
Section 2.1 (page unavailable) defines λ in a prior for bin boundaries, but does not describe how an optimal binning changes as λ increases. Sections 2.3–2.4 (pages unavailable) describe dynamic-programming searches, not a piecewise-linear solution path or merge-only breakpoints. Searches for those terms found no passage establishing the claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIf two adjacent bins are merged at some lambda in mPAVA, the new bin will never split for all larger lambda.
“The essential idea of mPAVA is based on a theorem stating that if two adjacent bins are merged on some value of λ to construct a larger bin, then the new bin will never split for all larger values of λ [25].”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [25]
The Methods sections on SBB and ABB describe Bayesian binning and dynamic-programming searches, but do not state a theorem about merged bins remaining intact as λ increases. A search for mPAVA and for adjacent-bin merges found no passage addressing that claim. Page numbers were not available in our copy. (LLM, conf=0.74)
Experimental — accuracy not yet measuredENIR uses mPAVA to find the solution path to a near isotonic regression problem in O(N log N), where N is the number of training instances.
“ENIR utilizes the path algorithm modified pool adjacent violators algorithm (mPAVA) that can find the solution path to a near isotonic regression problem in O(N log N), where N is the number of training instances [25].”
From Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models (2016) · cites this paper as [25]
The Discussion (page unavailable) identifies PAV as an isotonic-regression algorithm, but does not discuss ENIR or mPAVA. Searches for the near-isotonic solution path and the modified pool adjacent violators algorithm found no passage supporting the claimed O(N log N) complexity. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834 · see citing works
A Prediction Rule to Identify Low-Risk Patients with Community-Acquired Pneumonia
Fine, M. J., Auble, T. E., Yealy, D. M., Hanusa, B. H., Weissfeld, L. A., Singer, D. E., Coley, C. M., Marrie, T. J., & Kapoor, W. N. (1997) · New England Journal of Medicine · doi:10.1056/nejm199701233360402 · Cited by 4,560 · see citing works
Learning Bayesian Networks: The Combination of Knowledge and Statistical Data
Heckerman, D. E., Geiger, D., & Chickering, D. M. (1995) · Machine Learning · doi:10.1023/a:1022623210503 · Cited by 3,251 · see citing works
An empirical comparison of supervised learning algorithms
Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143865 · Cited by 2,780 · see citing works
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · see citing works
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
Taskar, B., Guestrin, C., & Koller, D. (2003) · Cited by 1,211 · see citing works
Statistical Inference under Order Restrictions (The Theory and Application of Isotonic Regression)
Vincze, I., Barlow, R. E., Bartholomew, D. J., Bremner, J. M., & Brunk, H. D. (1973) · International Statistical Review · doi:10.2307/1402630 · Cited by 840 · see citing works
(2001) · Cited by 683 · see citing works
The Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · see citing works
Zhu, J., Ahmed, A. A., & Xing, E. P. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553535 · Cited by 237 · see citing works
Calibrating predictive model estimates to support personalized medicine
Jiang, X., Osl, M., Kim, J., & Lucila Ohno-machado, A. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000291 · Cited by 209 · see citing works
Predicting accurate probabilities with a ranking loss
Menon, A. K., Jiang, X., Vembu, S., Elkan, C., & Lucila Ohno-machado, A. (2012) · PubMed · doi:10.48550/arxiv.1206.4661 · Cited by 62 · see citing works
Application of an efficient Bayesian discretization method to biomedical data
Lustgarten, J. L., Visweswaran, S., Gopalakrishnan, V., & Cooper, G. F. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-309 · Cited by 55 · see citing works
Self-citationAccurate probability calibration for multiple classifiers
Zhong, W., & Kwok, J. T.-Y. (2013) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 28 · see citing works
Laplace maximum margin Markov networks
Zhu, J., Xing, E. P., & Zhang, B. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390314 · Cited by 21 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [16]Zhu J Ahmed A Xing EP Medlda: maximum margin supervised topic models for regression and classification Proceedings of the 26th Annual International Conference on Machine Learning 1257 1264 2009
Cited by
Sorted by citations
On the appropriateness of Platt scaling in classifier calibration
Böken, B. (2021) · Information Systems · doi:10.1016/j.is.2020.101641 · Cited by 70 · see citing works
Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models
Naeini, M. P., & Cooper, G. F. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0047 · Cited by 46 · see citing works
“error (RMSE)6, maximum calibration error (MCE), and expected calibration error (ECE) [20], [21].”
BackgroundMethodInfluentialSelf-citation
- Authors
- Naeini, M. P., Cooper, G. F., & Hauskrecht, M.
- Affiliations
- University of Pittsburgh (US)
- Venue
- Proceedings of the 2015 SIAM International Conference on Data Mining
- Published
- 2015-06-30
- Volume
- 2015
- Pages
- 208-216
- PMC
- PMC4657569
- OpenAlex
- W2398242707
- Cited by
- 19
- References
- 17
- Citations by year
- 2026: 1 · 2024: 1 · 2023: 2 · 2022: 1 · 2021: 4 · 2020: 5 · 2019: 1 · 2017: 2 · 2016: 2
- FWCI
- 0.75
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- SIAM International Conference on Data Mining (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Science Foundation
- National Institutes of Health
- National Human Genome Research Institute (U54HG008540)
- National Institute of General Medical Sciences (R01GM088224)
- U.S. National Library of Medicine (R01LM010019)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- classifier calibration, binary classification, Bayesian model averaging, probabilistic forecasting, post-processing, discrimination and calibration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
