The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index; 14 of the 35 sources its claims cite were checked for support.
Classifier calibration: a survey on how to assess and improve predicted class probabilities
Open accessAbstract
This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.
Overview
CoChat Summary
TL;DR
This survey reviews the principles and practice of classifier calibration, covering definitions, proper scoring rules, evaluation metrics and visualisations, and post-hoc calibration methods for binary and multiclass classification. It aims to provide a unified perspective and practical guidance, illustrated with toy examples and synthetic test cases using the PyCalib library.
Key findings
- Multiclass calibration has at least three definitions: confidence calibration (only the highest predicted probability), classwise calibration (marginal probabilities), and multiclass calibration (the entire predicted probability vector); these are equivalent for binary classification but increasingly stronger for more than two classes.
- Proper scoring rules decompose into calibration loss and refinement loss; in the paper’s 10-instance, 3-class example, mean Brier score 0.71 = calibration loss 0.19 + refinement loss 0.52, and mean log-loss 1.12 = 0.29 + 0.83. A further decomposition gives Brier score 0.71 = epistemic loss 0.26 + irreducible loss 0.45, and log-loss 1.12 = 0.48 + 0.64.
- On the running 30-instance toy example, binary-ECE = 0.1873, classwise-ECE = 0.1787, classwise-MCE = 0.48, confidence-ECE = 0.2117, and confidence-MCE = 0.3; the Hosmer–Lemeshow test gives C = 25.3 with p = 0.0003, and a resampling test for classwise-ECE gives p ≈ 0.016.
- The survey details post-hoc methods including empirical binning, isotonic regression, Platt scaling, and Beta calibration for binary problems, and temperature, vector, matrix, and Dirichlet scaling for multiclass problems, discussing advantages and drawbacks such as temperature scaling’s single parameter preventing overfitting but being potentially suboptimal, and Dirichlet calibration supporting the identity map with ODIR regularisation.
Methods
- Design: a survey/tutorial-style overview that grew out of a 2020 ECML-PKDD tutorial, covering motivation, definitions, proper scoring rules, evaluation and visualisation, post-hoc calibration methods, hypothesis tests, and advanced topics.
- Data or sample: illustrative toy examples (a 30-instance, 3-class table and a 10-instance, 3-class table) and synthetic two-class and three-class test cases; the MLP test case uses a two-layer structure with 128 hidden units per layer, ReLU activation, and 10,000 sampled training instances.
- Main analysis: conceptual synthesis and worked calculations of calibration metrics and scoring-rule decompositions, plus empirical illustration of calibration maps and reliability diagrams; most figures and results were generated using PyCalib, not a systematic benchmark.
Limitations
- The paper states its focus is the classical setup where the classifier is deployed in the same setting where it was calibrated, except for potential changes in class prior or misclassification costs; it does not cover robustness under other distributional shifts or out-of-distribution inputs.
- The paper lists open problems: there is no standard method to decide binning type or number of bins; post-hoc calibration does not improve robustness against out-of-distribution inputs; and epistemic/second-order uncertainty remains future work.
- Evident from the text: comprehensive analysis of calibration in other research fields and applications is left as future work, and the empirical demonstrations are synthetic/toy examples rather than a broad real-world benchmark.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




















![Fig. 17 MLP classifier with Temperature scaling (top: class-wise calibration map, bottom: class-wise reliability diagram). While Temperature scaling manages to improve the reliability diagram around the [0.1, 0.4] region for classes 1 and 2, we can see it actually makes things worse elsewhere. As the calibration maps show, Temperature scaling applies the same calibration map to all three classes despite them having different class-wise reliability diagrams, which is the main reason for the sub-optimal results](https://figures.cochat.ai/paper-figures/f74e507f26524b81b8e769de4cd42b6b/classifier-calibration-a-survey-on-how-to-assess-and-improve-predicted-class-figure-17.png)





Verification
Claims the paper makes
9 Valid · 6 Not valid · 16 Not checked · 146 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The decomposition of proper scoring rule into calibration loss and refinement loss is then the sum of expected loss between S and C and expected loss between C and Y (Bröcker & Smith, 2007;Kull & Flach, 2015):”
§ Classifier calibration: a survey on how to assess and improve predicted class probabilities > 3.1 Decompositions of proper scoring rules
What the sources say
- Not checked — no full text of the sourcecites Bröcker & Smith, 2007 source
- Partly supportedcites Kull & Flach, 2015 sourceExperimental — accuracy not yet measured
The Abstract states that proper scoring rules have a well-known decomposition into calibration loss and refinement loss. It does not specify the claimed terms as expected loss between S and C and expected loss between C and Y, so that more precise formulation is not verified by our copy. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Some ideas do generalise quite easily: for example, we can model per-class scores by multivariate distributions such as Dirichlet distributions (Kull et al., 2019).”
§ Classifier calibration: a survey on how to assess and improve predicted class probabilities > 2.3 Multiclass calibration
What the sources say
- Supportedcites Kull et al., 2019 source
The Abstract states the method is 'derived from Dirichlet distributions and generalising the beta calibration method from binary classification', and Section 3 (Dirichlet calibration) does exactly what the citing sentence attributes: it considers the distribution of prediction vectors separately on instances of each class and takes these k distributions to be Dirichlet. The Conclusion reinforces the 'generalises easily' framing, calling Dirichlet calibration 'a natural extension of the two-class beta calibration method' and noting that deriving maps from Dirichlet distributions opens the door to other exponential-family distributions. One minor nuance not carried by the claim: the paper stresses the Dirichlet form is a derivation device rather than an assumption that the per-class vectors really are Dirichlet-distributed. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“One approach that works well is to model the classwise scores with Beta distributions (Kull et al., 2017a(Kull et al., , 2017b)).”
§ Classifier calibration: a survey on how to assess and improve predicted class probabilities > 2.2 Common forms of miscalibration, and how to fix them
What the sources say
- Supportedcites Kull et al., 2017a source
Section 3.1 (p. 9) derives beta calibration by assuming beta-distributed scores for each class. Section 3.2 (p. 10) shows that the resulting calibration map can correct probability estimates pushed toward the extremes, an example of overconfidence. The paper notes in Section 3.4 (p. 16) that good calibration does not require each class’s scores to fit a beta distribution exactly; modelling their distribution ratio can suffice. (LLM, conf=0.87)
- Partly supportedcites Kull et al., 2017b sourceExperimental — accuracy not yet measured
Section 3.1 (p. 9) derives beta calibration by assuming beta-distributed scores for each class, and Section 4.2 (p. 18) reports good performance for beta calibration. But Section 3.4 (pp. 15–16) distinguishes that method from fitting separate classwise beta distributions, which can fit poorly or produce inverted probabilities. The copy therefore supports beta calibration as an effective approach, not the unqualified claim that modelling each class’s scores with beta distributions works well for overconfidence specifically. (LLM, conf=0.83)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“In other words, binning approaches can reduce the calibration loss at the cost of a higher grouping loss (Kull & Flach, 2015).”
§ Classifier calibration: a survey on how to assess and improve predicted class probabilities > 5.3.1 Empirical binning
What the sources say
- Could not verifycites Kull & Flach, 2015 sourceExperimental — accuracy not yet measured
The available indexed passages are bibliographic metadata, not discussion of the paper’s findings. Searches for binning, calibration loss, and grouping loss returned no passage addressing the claimed trade-off, so our copy cannot verify it. (LLM, conf=0.72)
plainstates a causen = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“post-hoc calibration does not improve robustness against out-of-distribution inputs”
§ Classifier calibration: a survey on how to assess and improve predicted class probabilities > 7 Concluding remarks
What the sources say
- Partly supportedcites Ovadia et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says post-hoc calibration can fail under even mild input shift, and the MNIST results (p. 5) say calibration on a validation set does not guarantee calibration on shifted data. But the claim is categorical about out-of-distribution inputs: the genomic-sequences results (Appendix F, p. 24) report that temperature scaling performed better than the vanilla model on a test set mixing in-distribution and OOD data. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
63 more claims
Create a free account to see all 177 claims and to verify the 4 references still open.
- Probably not a substantial claim. 73
- It may state two separate claims as one. 55
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“60 to 69 15 11 75”
§ Classifier calibration: a survey on how to assess and improve predicted class probabilities > 2 A brief introduction to classifier calibration
Reported 75 · recomputed 73.333333
Citations
Cites
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Kupper, L. L., Hosmer, D. W., & Lemeshow, S. (1990) · Journal of the American Statistical Association · doi:10.2307/2290035 · Cited by 7,599
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,385
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Koenker, R., & Hallock, K. F. (2001) · Journal of Economic Perspectives · doi:10.1257/jep.15.4.143 · Cited by 4,413
(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Advances in Neural Information Processing Systems 27 (NIPS 2014)
Zhong, M., Goddard, N., & Sutton, C. A. (2014) · Cited by 2,636
Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,451
Research works printed but not linked: 3
- Bross, I. (1954). Design for decision. Science Education, 38(4), 325.
- Flach, P. (2014). Classification in context: Adapting to changes in class and cost distribution. In First international workshop on learning over multiple contexts at European conference on machine learning and principles and practice of knowledge discovery in databases (ECML-PKDD).
- Jiang, Y., Nagarajan, V., Baek, C., & Kolter, J. Z. (2021). Assessing generalization of sgd via disagreement. arXiv preprintarXiv:2106.13799.
Cited by
Automated machine learning: past, present and future
Baratchi, M., Wang, C., Limmer, S., van Rijn, J. N., Hoos, H. H., Bäck, T., & Olhofer, M. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10726-1 · Cited by 204
Methodology and evaluation in sports analytics: challenges, approaches, and lessons learned
Davis, J. J., Bransen, L., Devos, L., Jaspers, A., Meert, W., Robberechts, P., Haaren, J. V., & Roy, M. V. (2024) · Machine Learning · doi:10.1007/s10994-024-06585-0 · Cited by 151
“Indeed, many tasks in sports rely on having sufficiently well-calibrated probability estimates (Silva Filho et al., 2023).”
Resilience-aware MLOps for AI-based medical diagnostic system
Moskalenko, V., & Kharchenko, V. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1342937 · Cited by 32
Katsuki, M., Matsumori, Y., Kawamura, S., Kashiwagi, K., Koh, A., Tachikawa, S., & Yamagishi, F. (2023) · Headache The Journal of Head and Face Pain · doi:10.1111/head.14611 · Cited by 35
Human-in-the-loop active learning for goal-oriented molecule generation
Nahal, Y., Menke, J., Martinelli, J., Heinonen, M., Kabeshov, M., Janet, J. P., Nittinger, E., Engkvist, O., & Samuel Kaski, A. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-024-00924-y · Cited by 43
Metadata
- Authors
- Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P.
- Affiliations
- University of Bristol (GB)
- University of Tartu (EE)
- Venue
- Machine Learning
- Published
- 2023-05-16
- Volume
- 112
- Issue
- 9
- Pages
- 3211-3260
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W4376891107
- Semantic Scholar
- f471af0ba7d366233da32a1f7151fde42816790f
- Cited by
- 228
- References
- 95
- Citations by year
- 2026: 119 · 2025: 56 · 2024: 38 · 2023: 15
- FWCI
- 21.43
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Machine Learning (publishedVersion, cc-by)
- Bristol Research (University of Bristol) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Greater South Information System
- Funders
- Engineering and Physical Sciences Research Council (EP/R005273/1)
- Alan Turing Institute (EP/N510129/1)
- Estonian Research Competency Council (PRG1604)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- classifier calibration, uncertainty quantification, proper scoring rules, post-hoc calibration, multi-class classification, binary classification, probability calibration, cost-sensitive classification, decision making under uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Classifier calibration: a survey on how to assess and improve predicted class probabilities. https://app.cochat.ai/papers/10.1007/s10994-023-06336-7
- The paper
- Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P. (2023). Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning. https://doi.org/10.1007/s10994-023-06336-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.