CoChat
Sign Up
Cited by 46Imbalanced Data Classification TechniquesMachine Learning and Data ClassificationMachine Learning and Algorithms

Binary Classifier Calibration Using an Ensemble of Near Isotonic Regression Models

Open access
Naeini, Mahdi Pakdaman, Cooper, Gregory F.·2016·2016 IEEE 16th International Conference on Data Mining (ICDM)
DOI 10.1109/icdm.2016.0047PMC PMC5351887
Open this paper in CoChat — free account

Abstract

Learning accurate probabilistic models from data is crucial in many practical tasks in data mining. In this paper we present a new non-parametric calibration method called Ensemble of Near Isotonic Regression (ENIR). The method can be considered as an extension of BBQ, a recently proposed calibration method, as well as the commonly used calibration method based on isotonic regression (IsoRegC). ENIR is designed to address the key limitation of IsoRegC which is the monotonicity assumption of the predictions. Similar to BBQ, the method post-processes the output of a binary classifier to obtain calibrated probabilities. Thus it can be used with many existing classification models to generate accurate probabilistic predictions. We demonstrate the performance of ENIR on synthetic and real datasets for commonly applied binary classification models. Experimental results show that the method outperforms several common binary classifier calibration methods. In particular on the real data, ENIR commonly performs statistically significantly better than the other methods, and never worse. It is able to improve the calibration power of classifiers, while retaining their discrimination power. The method is also computationally tractable for large scale datasets, as it is O(N log N) time, where N is the number of samples.

Integrity signals

Registry

Crossref · 10.1109/icdm.2016.0047

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 13 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    If two adjacent bins are merged at some lambda in mPAVA, the new bin will never split for all larger lambda.

    “The essential idea of mPAVA is based on a theorem stating that if two adjacent bins are merged on some value of λ to construct a larger bin, then the new bin will never split for all larger values of λ [25].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] source

      The Methods sections on SBB and ABB describe Bayesian binning and dynamic-programming searches, but do not state a theorem about merged bins remaining intact as λ increases. A search for mPAVA and for adjacent-bin merges found no passage addressing that claim. Page numbers were not available in our copy. (LLM, conf=0.74)

      Experimental — accuracy not yet measured
  • Premisecitation

    ENIR uses mPAVA to find the solution path to a near isotonic regression problem in O(N log N), where N is the number of training instances.

    “ENIR utilizes the path algorithm modified pool adjacent violators algorithm (mPAVA) that can find the solution path to a near isotonic regression problem in O(N log N), where N is the number of training instances [25].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] source

      The Discussion (page unavailable) identifies PAV as an isotonic-regression algorithm, but does not discuss ENIR or mPAVA. Searches for the near-isotonic solution path and the modified pool adjacent violators algorithm found no passage supporting the claimed O(N log N) complexity. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Premisecitation

    The near-isotonic regression problem yields a unique solution p-hat-lambda for a given lambda.

    “It yields a unique solution p̂λ, where the use of the subscript λ emphasizes the dependency of the final solution to the value of λ.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] source

      The Introduction and Discussion discuss isotonic regression and relaxing its monotonicity constraint, but do not address whether a near-isotonic regression problem has a unique solution. In Methods §2.1, λ is the mean parameter of a Poisson prior on bin boundaries, not a parameter of a near-isotonic solution. Searches for near-isotonic regression and a unique solution for a given λ found no supporting passage in our copy; page numbers were unavailable. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    ENIR's near-isotonic assumption is realistic for classifier outputs.

    “Our experiments show that the near-isotonic assumption made by ENIR is a realistic assumption about the output of classifiers”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ENIR assumes that the mapping from uncalibrated scores to calibrated probabilities is near isotonic, allowing violations of classifier ordering that are penalized by a regularization term.
  • Findingnone given

    ENIR is computationally tractable for large scale datasets, running in O(N log N) time where N is the number of samples.

    “The method is also computationally tractable for large scale datasets, as it is O(N log N) time, where N is the number of samples.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 25
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 5

  • May combine two claims.

    Premisecitation

    mPAVA finds the entire solution path for near isotonic regression in O(N log N) time and O(N) memory.

    “mPAVA finds the whole solution path in O(N log N), and needs O(N) memory space.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] source

      The Discussion (page unavailable) mentions the PAV algorithm and gives a running time for isotonic regression, but does not discuss mPAVA or a near-isotonic solution path. Searches for mPAVA, near-isotonic regression, solution-path complexity, and memory use found no passage supporting the claimed O(N log N) time or O(N) space bounds. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Interpretationcitation

    As λ grows, bins can only merge and cannot split, so the solution path is piecewise linear and its breakpoints occur when two bins merge.

    “Consequently, the solution path remains piecewise linear as λ increases, and the breakpoints happen when two bins merge together.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] source

      Section 2.1 (page unavailable) defines λ in a prior for bin boundaries, but does not describe how an optimal binning changes as λ increases. Sections 2.3–2.4 (pages unavailable) describe dynamic-programming searches, not a piecewise-linear solution path or merge-only breakpoints. Searches for those terms found no passage establishing the claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    On real data, ENIR commonly performs statistically significantly better than other calibration methods and never worse.

    “In particular on the real data, ENIR commonly performs statistically significantly better than the other methods, and never worse.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ENIR is empirically shown to improve calibration of LR, SVM, and NB classifiers while maintaining their discrimination performance.

    “The method we describe in this paper is shown empirically to improve calibration of different types of classifiers (e.g., LR, SVM, and NB) while maintaining their discrimination performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ENIR is empirically shown to improve calibration of NB models without reducing their discrimination.

    “The method we describe is shown empirically to improve the calibration of NB models without reducing their discrimination.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Post-processing classifier outputs with ENIR yields high calibration improvement without losing statistically meaningful discrimination performance.

    “Moreover, our experiments show that by post processing the output of classifiers using ENIR, we can gain high calibration improvement, without losing any statistically meaningful discrimination performance.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    In experiments on synthetic and real datasets, ENIR outperforms several common binary classifier calibration methods.

    “Experimental results show that the method outperforms several common binary classifier calibration methods.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ENIR improves the calibration power of classifiers while retaining their discrimination power.

    “It is able to improve the calibration power of classifiers, while retaining their discrimination power.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For RMSE, ENIR is statistically superior to BBQ but not statistically different from IsoRegC.

    “The result indicates that in terms of RMSE, ENIR is statistically superior to BBQ; however, it is not performing statistically differently than IsoRegC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Overall, for calibration measured by ECE, MCE, and RMSE, ENIR either outperforms the other calibration methods or is statistically equivalent to IsoRegC and BBQ.

    “In terms of calibration measured by ECE, MCE, and RMSE, ENIR either outperforms other calibration methods, or it has a statistically equivalent performance to IsoRegC and BBQ.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Post-processing SVM outputs with ENIR gives a 95% confidence interval for AUC loss of at most 1%, but because the interval includes zero the AUC difference is not statistically significant.

    “This could be a significant improvement, depending on the application, considering the 95% CI for the AUC which shows that by using ENIR we are 95% confident not to lose more than 1% of the SVM discrimination power in terms of AUC (Note, however, that the CI includes zero, which indicates that there is not a statistically significant difference between the performance of SVM and ENIR in terms of 𝐴”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    On the simulated dataset, ENIR generally performs better than IsoRegC, particularly when the base learner is a linear SVM.

    “As seen in Table I, ENIR generally outperforms IsoRegC on the simulation dataset, especially when the linear SVM method is used as the base learner.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With SVM, ENIR is superior to IsoRegC and BBQ in AUC and ACC, but those differences are not statistically significant.

    “The results show that although ENIR is superior to IsoRegC and BBQ in terms of discrimination measures, AUC and ACC, the difference is not statistically significant.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With LR as the base classifier, ENIR is always superior to BBQ and IsoRegC except for MCE, where BBQ is superior but the difference is not statistically significant.

    “Table IV shows the results of comparison when we use LR as the base classifier. As shown, the performance of ENIR is always superior to BBQ and IsoRegC except for MCE in which BBQ is superior to ENIR; however, this difference is not statistically significant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Experiments show that post-processing classifier outputs with ENIR gives large calibration improvements in RMSE, ECE, and MCE without losing statistically meaningful discrimination performance.

    “Our experiments show that by post processing the output of classifiers using ENIR, we can gain high calibration improvement in terms of RMSE, ECE, and MCE, without losing any statistically meaningful discrimination performance.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The transformation from a classifier's uncalibrated output to calibrated probabilities is approximately monotonic, though not necessarily exactly monotonic.

  • May combine two claims.

    Findingown data

    With SVM, ENIR is statistically superior to BBQ in RMSE and to IsoRegC in MCE.

    “In terms of calibration measures, ENIR performs statistically superior to BBQ in terms of RMSE and it is statistically superior to IsoRegC in terms of MCE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With NB, ENIR is statistically superior to BBQ in ECE and RMSE.

    “ENIR is also statistically superior to BBQ in terms of ECE and RMSE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    With SVM, ENIR is always superior to BBQ and IsoRegC except for MCE, where BBQ performs better but the difference is not statistically significant.

    “Table V shows the results when we use SVM as the base classifier. As shown, the performance of ENIR is always superior to BBQ and IsoRegC except for MCE in which BBQ performs better than ENIR; however, the difference is not statistically significant for MCE.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    With NB, ENIR shows no statistically significant AUC difference from BBQ and IsoRegC, but is statistically superior to BBQ in ACC.

    “In terms of discrimination, for AUC there is not a statistically significant difference between the performance of ENIR compared with BBQ and IsoRegC; however, in terms of ACC, ENIR is statistically superior to BBQ.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Overall, for discrimination measured by AUC and ACC, ENIR either outperforms IsoRegC and BBQ or is not statistically significantly different from them.

    “Overall, in terms of discrimination measured by AUC and ACC, the results show that the proposed calibration method either outperforms IsoRegC and BBQ, or has a performance that is not statistically significantly different.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Deviations from perfect calibration are common in practice and vary widely depending on the binary classification model used.

    “Deviations from perfect calibration are very common in practice and may vary widely depending on the binary classification model that is used [20].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Isotonic regression calibration requires O(N log N) computations because instances must be sorted by classifier scores.

    “Note, however, that isotonic regression calibration still needs O(N log N) computations due to the fact that instances are required to be sorted based on the classifier scores yi.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ABB addresses BBQ's binning limitation by Bayesian averaging over all binning models induced by training instances.

    “A Bayesian non-parametric method called ABB addresses the latter problem by considering Bayesian averaging over all possible binning models induced by the training instances [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationnone given

    ENIR is computationally tractable and runs in O(N log N) for N training instances.

    “The method is computationally tractable, as it runs in O(N log N) for N training instances.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BBQ addresses histogram binning drawbacks by considering multiple equal frequency histogram binning models and combining them with a Bayesian scoring function.

    “BBQ addresses the main drawbacks of the histogram binning model by considering multiple different equal frequency histogram binning models and their combination using a Bayesian scoring function [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An isotonic regression-based calibration can be viewed as a histogram binning method where boundaries are selected by fitting the best monotone approximation to training data according to classifier ranking.

    “Therefore, an isotonic regression-based calibration can be viewed as a histogram binning method [27] where the position of boundaries are selected by fitting the best monotone approximation to the training data according to the ranking imposed by the classifier.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    ABB is computationally intractable for most real world applications, requiring O(N^2) computations for learning and O(N^2) for each test instance.

    “The main drawback of ABB is that it is computationally intractable for most real world applications, as it requires O(N2) computations for learning the model as well as O(N2) computations for computing the calibrated estimate for each of the test instances1.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    ENIR is an ensemble method that uses BIC to perform selective model averaging over near isotonic regression models, and IsoRegC is one extreme member of that set.

    “Second, ENIR is an ensemble model that utilizes the BIC scoring function to perform selective model averaging over a set of near isotonic regression models that indeed includes IsoRegC as an extreme member.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Each model in that family can be interpreted as a histogram binning model whose bin boundaries and sizes are chosen by balancing fit to the data against keeping the base classifier's ranking, with λ controlling that balance; larger λ favors ranking preservation and smaller λ favors fit.

    “Each of these models can be considered as a histogram binning model where the position of boundaries and the size of bins are selected according to how well the model trades off the goodness of fit with the preservation of the ranking generated by the classifier, which is governed by the value of λ”
    • Also stated in: § III. Method · plain
      “(As λ increases the model is more concerned to preserving the original ranking of the classifier, while for the small λ it prioritizes the goodness of fit.)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The second constraint in the isotonic regression optimization is redundant and the equation can be rewritten in an equivalent form.

    “One can show that the second constraint in the optimization given by Equation 1 is redundant, and it is possible to rewrite the equation in the following equivalent form:”
    Quote in the paperNot a substantial claim
30 of 30

  • The authors state this as an assumption.

    ENIR assumes that the mapping from uncalibrated scores to calibrated probabilities is near isotonic, allowing violations of classifier ordering that are penalized by a regularization term.

    “ENIR assumes that the mapping from uncalibrated scores to calibrated probabilities is a near isotonic (monotonic) mapping; it allows violations of the ordering imposed by the classifier and then penalizes them through the use of a regularization term.”

  • The authors state this as an assumption.

    IsoRegC assumes that the mapping from classifier output to calibrated probability is isotonic and follows the ranking imposed by the base classifier.

    “To build a mapping from the uncalibrated output of a classifier to the calibrated probability, IsoRegC assumes the mapping is an isotonic (monotonic) mapping following the ranking imposed by the base classifier.”

  • The authors state this as an assumption.

    The rationale behind standard isotonic regression calibration assumes that the base classifier ranks instances correctly.

    “The rationale behind this model is to assume that the base classifier ranks the instances correctly.”

  • The authors state this as an assumption.

    ENIR uses the prior knowledge that the scores to be calibrated come from a classifier with good discrimination performance.

    “The essential idea in ENIR is to use the prior knowledge that the scores to be calibrated are in fact generated by a well-performing classifier in terms of discrimination.”

  • The paper relies on this without support.

    Research on learning well-calibrated models has not been explored in the data mining literature as extensively as learning models with high discrimination.

    “Research on learning well calibrated models has not been explored in the data mining literature as extensively as, for example, learning models that have high discrimination (e.g., high accuracy).”

  • The paper relies on this without support.

    Post-processing calibration can decrease discrimination while increasing calibration if care is not taken.

    “However, this approach has the potential to decrease discrimination while increasing calibration, if care is not taken.”

  • The authors state this as an assumption.

    The transformation from a classifier's uncalibrated output to calibrated probabilities is approximately monotonic, though not necessarily exactly monotonic.

    “ENIR makes a more realistic assumption compared to IsoRegC by assuming that the transformation from the uncalibrated output of a classifier to calibrated probability estimates is approximately (but not necessarily exactly) a monotonic function.”

  • The authors state this as an assumption.

    The optimal predicted value for each bin in mPAVA is well-defined and satisfies the zero partial derivative condition.

    “Assuming that p̂Bi (λ) is optimal, the partial derivative of ℒB,λ has to be 0 at p̂Bi(λ), which implies:”

  • The authors state this as an assumption.

    The two bins Bi and Bi+1 are the first two bins to merge as λ increases.

    “assuming that the two bins Bi and Bi+1 are the first two bins to merge by increasing λ, the value of λi,i+1 at which the two bins Bi and Bi+1 will merge is calculated as:”
9 of 9

Cites

Flagged first, then by citations

5 of the 31 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. hashemi, 2010, Nasser Yazdani, Azadeh Shakery, and Mahdi Pakdaman Naeini. Application of ensemble models in web ranking, Proc International Symposium on Telecommunication (IST), 726
  2. jennifer, 1999, Bayesian model averaging: a tutorial, Statistical Science, 382
  3. zhang, 2004, Naïve Bayesian classifiers for ranking, Proceedings of the European Conference on Machine Learning (ECML), 501
  4. 2004, Ira Cohen and Moises Goldszmidt. Properties and benefits of calibrated classifiers, Proceedings of the European Conference on Principles of Data Mining and Knowledge Discovery, 125
  5. philip, 1981, Practical Optimization
  6. russell, 2010, Artificial Intelligence A Modern Approach

Cited by

Sorted by citations

Related

Authors
Naeini, M. P., & Cooper, G. F.
Affiliations
University of Pittsburgh (US)
ORCID
Venue
2016 IEEE 16th International Conference on Data Mining (ICDM)
Publisher
IEEE
Published
2016-12
Volume
2016
Pages
360-369
ISSN
1550-4786 (ISSN-L)
DOI
10.1109/icdm.2016.0047
PMC
PMC5351887
OpenAlex
W2273082833
Cited by
46
References
31
Citations by year
2026: 3 · 2025: 6 · 2024: 6 · 2023: 6 · 2022: 3 · 2021: 8 · 2020: 9 · 2019: 2 · 2017: 3
FWCI
0.90
Type
conference paper
Language
en
Open access
Locations
  • Proceedings (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Pennsylvania Department of Health
  • National Institutes of Health
  • National Human Genome Research Institute (U54HG008540)
  • National Institute of General Medical Sciences (R01GM088224)
  • U.S. National Library of Medicine (R01LM012095)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
probabilistic classification, isotonic regression, non-parametric calibration, monotonicity assumption, discriminatory power

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review