Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index; 14 of the 35 sources its claims cite were checked for support.

Classifier calibration: a survey on how to assess and improve predicted class probabilities

Open access
Silva Filho, Telmo, Song, Hao, Perello-Nieto, Miquel, Santos-Rodriguez, Raul, Kull, Meelis, Flach, Peter · 2023 · Machine Learning

Abstract

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

Overview

CoChat Summary

TL;DR

This survey reviews the principles and practice of classifier calibration, covering definitions, proper scoring rules, evaluation metrics and visualisations, and post-hoc calibration methods for binary and multiclass classification. It aims to provide a unified perspective and practical guidance, illustrated with toy examples and synthetic test cases using the PyCalib library.

Key findings

  • Multiclass calibration has at least three definitions: confidence calibration (only the highest predicted probability), classwise calibration (marginal probabilities), and multiclass calibration (the entire predicted probability vector); these are equivalent for binary classification but increasingly stronger for more than two classes.
  • Proper scoring rules decompose into calibration loss and refinement loss; in the paper’s 10-instance, 3-class example, mean Brier score 0.71 = calibration loss 0.19 + refinement loss 0.52, and mean log-loss 1.12 = 0.29 + 0.83. A further decomposition gives Brier score 0.71 = epistemic loss 0.26 + irreducible loss 0.45, and log-loss 1.12 = 0.48 + 0.64.
  • On the running 30-instance toy example, binary-ECE = 0.1873, classwise-ECE = 0.1787, classwise-MCE = 0.48, confidence-ECE = 0.2117, and confidence-MCE = 0.3; the Hosmer–Lemeshow test gives C = 25.3 with p = 0.0003, and a resampling test for classwise-ECE gives p ≈ 0.016.
  • The survey details post-hoc methods including empirical binning, isotonic regression, Platt scaling, and Beta calibration for binary problems, and temperature, vector, matrix, and Dirichlet scaling for multiclass problems, discussing advantages and drawbacks such as temperature scaling’s single parameter preventing overfitting but being potentially suboptimal, and Dirichlet calibration supporting the identity map with ODIR regularisation.

Methods

  • Design: a survey/tutorial-style overview that grew out of a 2020 ECML-PKDD tutorial, covering motivation, definitions, proper scoring rules, evaluation and visualisation, post-hoc calibration methods, hypothesis tests, and advanced topics.
  • Data or sample: illustrative toy examples (a 30-instance, 3-class table and a 10-instance, 3-class table) and synthetic two-class and three-class test cases; the MLP test case uses a two-layer structure with 128 hidden units per layer, ReLU activation, and 10,000 sampled training instances.
  • Main analysis: conceptual synthesis and worked calculations of calibration metrics and scoring-rule decompositions, plus empirical illustration of calibration maps and reliability diagrams; most figures and results were generated using PyCalib, not a systematic benchmark.

Limitations

  • The paper states its focus is the classical setup where the classifier is deployed in the same setting where it was calibrated, except for potential changes in class prior or misclassification costs; it does not cover robustness under other distributional shifts or out-of-distribution inputs.
  • The paper lists open problems: there is no standard method to decide binning type or number of bins; post-hoc calibration does not improve robustness against out-of-distribution inputs; and epistemic/second-order uncertainty remains future work.
  • Evident from the text: comprehensive analysis of calibration in other research fields and applications is left as future work, and the empirical demonstrations are synthetic/toy examples rather than a broad real-world benchmark.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10994-023-06336-7

Cites

103 of 103 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY · From the PDF
Fig. 2 Dividing the forecasts in Table 1 in five (left figure) and two (right figure) equal-width bins
Fig. 2 Dividing the forecasts in Table 1 in five (left figure) and two (right figure) equal-width binsCC-BY · From the PDF
Fig. 3 Examples of under- and overconfident classifiers. The axes are the same as for the previous figures: x-axis shows predicted probabilities and y-axis shows empirical probabilities. The dots represent the reliability diagram, and the lines show the best-fit logistic curve and beta curve. Note that the logistic sigmoid is a good fit for underconfident scores but not for overconfidence. The figure has been adapted from Niculescu-Mizil and Caruana (2005), and the beta curves have been added
Fig. 3 Examples of under- and overconfident classifiers. The axes are the same as for the previous figures: x-axis shows predicted probabilities and y-axis shows empirical probabilities. The dots represent the reliability diagram, and the lines show the best-fit logistic curve and beta curve. Note that the logistic sigmoid is a good fit for underconfident scores but not for overconfidence. The figure has been adapted from Niculescu-Mizil and Caruana (2005), and the beta curves have been addedCC-BY · From the PDF
Image 4
CC-BY · From the PDF
Fig. 4 Graphical illustration of how a well-fitted calibration map leads to near-perfect calibration. From left to right, the dotted arrows show (1) using a point's uncalibrated score s on the x -axis as input to the calibration map, (2) mapping the resulting output u1D5BC ( s ) back to the diagonal, and (3) combine with the empirical probability of the point we started from. The closer the original point is to the fitted calibration map, the closer the calibrated point (in red) will be to the diagonal
Fig. 4 Graphical illustration of how a well-fitted calibration map leads to near-perfect calibration. From left to right, the dotted arrows show (1) using a point's uncalibrated score s on the x -axis as input to the calibration map, (2) mapping the resulting output u1D5BC ( s ) back to the diagonal, and (3) combine with the empirical probability of the point we started from. The closer the original point is to the fitted calibration map, the closer the calibrated point (in red) will be to the diagonalCC-BY · From the PDF
Fig. 5 Different graphical representations of calibration-related information found in the literature
Fig. 5 Different graphical representations of calibration-related information found in the literatureCC-BY · From the PDF
(a) Not calibrated example. 4 samples with score 0 from which 25% are positive, and 4 samples with score 1 from which 75% are positive.
(a) Not calibrated example. 4 samples with score 0 from which 25% are positive, and 4 samples with score 1 from which 75% are positive.CC-BY · From the PDF
(b) Calibrated example. 1 sample with score 0 from negative class, and 1 sample with score 1 from positive class.
(b) Calibrated example. 1 sample with score 0 from negative class, and 1 sample with score 1 from positive class.CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Fig. 8 Reliability diagram corresponding to the 5 bins presented in Table 4
Fig. 8 Reliability diagram corresponding to the 5 bins presented in Table 4CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Fig. 10 Confidence reliability diagram corresponding to the bins in Table 7
Fig. 10 Confidence reliability diagram corresponding to the bins in Table 7CC-BY · From the PDF
Fig. 11 The synthetic dataset used in the test cases. The left two figures shows the data distribution for each class respectively. The figure on the right gives the Bayes-optimal posterior probability for class 1
Fig. 11 The synthetic dataset used in the test cases. The left two figures shows the data distribution for each class respectively. The figure on the right gives the Bayes-optimal posterior probability for class 1CC-BY · From the PDF
Fig. 12 The reliability diagrams of the three uncalibrated classifiers. The left figure shows the ̂ u1D5BF under case, where the classifier predicts a probability closer to 0.5 compared with the Bayes-optimal case (Fig. 11, right), resulting in a reliability diagram with steeper slope. The middle figure shows the ̂ u1D5BF over case, where the predicted probabilities are pushed towards 0 and 1 respectively, resulting in a shallower reliability curve. The right figure is obtained with a MLP classifier ( ̂ u1D5BF MLP ), where the reliability diagram shows over-confident and under-confident predictions around different regions
Fig. 12 The reliability diagrams of the three uncalibrated classifiers. The left figure shows the ̂ u1D5BF under case, where the classifier predicts a probability closer to 0.5 compared with the Bayes-optimal case (Fig. 11, right), resulting in a reliability diagram with steeper slope. The middle figure shows the ̂ u1D5BF over case, where the predicted probabilities are pushed towards 0 and 1 respectively, resulting in a shallower reliability curve. The right figure is obtained with a MLP classifier ( ̂ u1D5BF MLP ), where the reliability diagram shows over-confident and under-confident predictions around different regionsCC-BY · From the PDF
(c) Test case 3: MLP classifier
(c) Test case 3: MLP classifierCC-BY · From the PDF
(c) Test case 3: MLP classifier
(c) Test case 3: MLP classifierCC-BY · From the PDF
Image 19
CC-BY · From the PDF
Image 20
CC-BY · From the PDF
Fig. 17 MLP classifier with Temperature scaling (top: class-wise calibration map, bottom: class-wise reliability diagram). While Temperature scaling manages to improve the reliability diagram around the [0.1, 0.4] region for classes 1 and 2, we can see it actually makes things worse elsewhere. As the calibration maps show, Temperature scaling applies the same calibration map to all three classes despite them having different class-wise reliability diagrams, which is the main reason for the sub-optimal results
Fig. 17 MLP classifier with Temperature scaling (top: class-wise calibration map, bottom: class-wise reliability diagram). While Temperature scaling manages to improve the reliability diagram around the [0.1, 0.4] region for classes 1 and 2, we can see it actually makes things worse elsewhere. As the calibration maps show, Temperature scaling applies the same calibration map to all three classes despite them having different class-wise reliability diagrams, which is the main reason for the sub-optimal resultsCC-BY · From the PDF
Fig. 18 Class-wise calibration maps and reliability diagrams for the MLP classifier with Vector scaling. The most obvious change from Temperature scaling is that now the calibration maps are different for each class. However, the calibration maps still have a circular shape as Vector scaling only adjusts one dimension of the logit vector for each class. As a result, the reliability diagram is close to the ones of Temperature scaling. Some minor improvements can be seen with class 3, where Vector scaling does not make it quite as bad as for Temperature scaling
Fig. 18 Class-wise calibration maps and reliability diagrams for the MLP classifier with Vector scaling. The most obvious change from Temperature scaling is that now the calibration maps are different for each class. However, the calibration maps still have a circular shape as Vector scaling only adjusts one dimension of the logit vector for each class. As a result, the reliability diagram is close to the ones of Temperature scaling. Some minor improvements can be seen with class 3, where Vector scaling does not make it quite as bad as for Temperature scalingCC-BY · From the PDF
Fig. 19 Class-wise calibration maps and reliability diagrams for the MLP classifier with Matrix scaling. As Matrix scaling is capable of providing fully linear adjustments on the logits for each class, we can see the class-wise calibration map no longer has the circular shape. While there are still minor improvements on the reliability diagram of class 1 and class 2, class 3 has gained a significant improvement
Fig. 19 Class-wise calibration maps and reliability diagrams for the MLP classifier with Matrix scaling. As Matrix scaling is capable of providing fully linear adjustments on the logits for each class, we can see the class-wise calibration map no longer has the circular shape. While there are still minor improvements on the reliability diagram of class 1 and class 2, class 3 has gained a significant improvementCC-BY · From the PDF
Fig. 20 Class-wise calibration maps and reliability diagrams for the MLP classifier with Dirichlet calibration. The overall results on the class-wise reliability diagrams are close to those obtained with Matrix scaling. The major differences are seen in the calibration maps, where we can clearly observe a local region for the changes on class 3. Importantly, this region for the Dirichlet calibration approach doesn't cover the area where the predicted probability of class 3 is close to 1. This is more reasonable compared with Matrix scaling, as both the Bayes-optimal predictions and the MLP predictions for class 3 are capped around 0.8
Fig. 20 Class-wise calibration maps and reliability diagrams for the MLP classifier with Dirichlet calibration. The overall results on the class-wise reliability diagrams are close to those obtained with Matrix scaling. The major differences are seen in the calibration maps, where we can clearly observe a local region for the changes on class 3. Importantly, this region for the Dirichlet calibration approach doesn't cover the area where the predicted probability of class 3 is close to 1. This is more reasonable compared with Matrix scaling, as both the Bayes-optimal predictions and the MLP predictions for class 3 are capped around 0.8CC-BY · From the PDF
Image 25
CC-BY · From the PDF
Image 26
CC-BY · From the PDF

Verification

Claims the paper makes

9 Valid · 6 Not valid · 16 Not checked · 146 Own results and interpretations

63 more claims

Create a free account to see all 177 claims and to verify the 4 references still open.

See all claims — free account
  • Probably not a substantial claim. 73
  • It may state two separate claims as one. 55

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 95 references

Research works printed but not linked: 3

  1. Bross, I. (1954). Design for decision. Science Education, 38(4), 325.
  2. Flach, P. (2014). Classification in context: Adapting to changes in class and cost distribution. In First international workshop on learning over multiple contexts at European conference on machine learning and principles and practice of knowledge discovery in databases (ECML-PKDD).
  3. Jiang, Y., Nagarajan, V., Baek, C., & Kolter, J. Z. (2021). Assessing generalization of sgd via disagreement. arXiv preprintarXiv:2106.13799.

Cited by

See all 228

Metadata

Authors
Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P.
Affiliations
  • University of Bristol (GB)
  • University of Tartu (EE)
Venue
Machine Learning
Published
2023-05-16
Volume
112
Issue
9
Pages
3211-3260
ISSN
  • 0885-6125 (ISSN-L)
  • 1573-0565
OpenAlex
W4376891107
Cited by
228
References
95
Citations by year
2026: 119 · 2025: 56 · 2024: 38 · 2023: 15
FWCI
21.43
Type
journal article
Language
en
Open access
Licence
Locations
  • Machine Learning (publishedVersion, cc-by)
  • Bristol Research (University of Bristol) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Explore Bristol Research (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
  • Greater South Information System
Funders
  • Engineering and Physical Sciences Research Council (EP/R005273/1)
  • Alan Turing Institute (EP/N510129/1)
  • Estonian Research Competency Council (PRG1604)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
classifier calibration, uncertainty quantification, proper scoring rules, post-hoc calibration, multi-class classification, binary classification, probability calibration, cost-sensitive classification, decision making under uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Classifier calibration: a survey on how to assess and improve predicted class probabilities. https://app.cochat.ai/papers/10.1007/s10994-023-06336-7
The paper
Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P. (2023). Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning. https://doi.org/10.1007/s10994-023-06336-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.