CoChat
Sign Up
PreprintCited by 3,447Adversarial Robustness in Machine LearningAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 49 of the 49 references it cites were found in a registry or an index; 2 of the 7 sources its claims cite were checked for support.

On Calibration of Modern Neural Networks

Open access
Guo, Chuan, Pleiss, Geoff, Sun, Yu, Weinberger, Kilian Q.·2017·arXiv
DOI 10.48550/arxiv.1706.04599arXiv 1706.04599
Open this paper in CoChat — free accountOpen PDF

Abstract

Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

Modern neural networks are often poorly calibrated: their predicted confidence exceeds their accuracy, unlike older networks. Temperature scaling, a single-parameter post-processing method, is surprisingly effective at fixing this across vision and NLP classification tasks.

Key findings

  • Miscalibration is common in modern architectures: on CIFAR-100, a 110-layer ResNet has ECE 16.53% uncalibrated, while a 5-layer LeNet has ECE 4.85%; most datasets/models show ECE typically between 4 to 10%, with exceptions SVHN at 0.44% and Reuters at 0.85%.
  • Depth, width, Batch Normalization, and weight decay affect calibration: increased model capacity and less weight decay are associated with worse ECE, and a 6-layer ConvNet with Batch Normalization has worse calibration even when accuracy improves slightly.
  • Temperature scaling is often the most effective method: on CIFAR-100 ResNet 110 it reduces ECE from 16.53% to 1.26%, and on CIFAR-10 ResNet 110 from 4.6% to 0.83%; it outperforms vector and matrix scaling on vision tasks and performs comparably on NLP datasets.
  • NLL can overfit after classification accuracy improves: on CIFAR-100, test error drops from 29% to 27% in the region where NLL overfits, suggesting networks gain accuracy at the expense of well-modeled probabilities.

Methods

  • The paper evaluates post-processing calibration methods: histogram binning, isotonic regression, Bayesian Binning into Quantiles (BBQ), and Platt scaling variants including temperature scaling, vector scaling, and matrix scaling.
  • Experiments cover 6 image datasets (Caltech-UCSD Birds, Stanford Cars, ImageNet 2012, CIFAR-10/100, SVHN) and 4 document/sentiment datasets (20 News, Reuters, SST binary, SST fine-grained), using ResNets, stochastic-depth ResNets, Wide ResNets, DenseNets, Deep Averaging Networks, and TreeLSTMs.
  • Calibration is measured primarily by Expected Calibration Error (ECE) with M = 15 bins, with additional MCE, NLL, and test error tables; temperature scaling uses a single scalar T > 0 applied to logits and optimized by NLL on a validation set, leaving class predictions unchanged.

Limitations

  • The paper states it cannot claim causality for the observed relationships between architecture/training trends and miscalibration, and says it remains future work to understand why these trends affect calibration while improving accuracy.
  • Temperature scaling does not calibrate the Reuters dataset; the authors note this dataset is already well-calibrated (ECE ≤ 1%) and that measurements may be affected by dataset split or binning scheme.
  • Matrix scaling performs poorly on datasets with hundreds of classes and fails to converge on 1000-class ImageNet, which the authors attribute to its number of parameters scaling quadratically with classes and overfitting a small validation set.
  • The supplement states MCE is very sensitive to the binning scheme and less suited for small test sets.
  • Evident from the text: all calibration methods require a hold-out validation set and assume training, validation, and test sets are drawn from the same distribution; evaluation is limited to the classification datasets and architectures listed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.04599

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 of 7 cited sources checked · 2 supported · 0 flagged

  • Probably not a substantial claim. 14
  • It may state two separate claims as one. 22
  • The quote only partly states this claim. 1
33 of 33

  • The authors state this as an assumption.

    The training, validation, and test sets are drawn from the same distribution.

    “We assume that the training, validation, and test sets are drawn from the same distribution.”

  • The authors state this as an assumption.

    There is no prior use of temperature scaling in the context of calibrating probabilistic models.

    “To the best of our knowledge, we are not aware of any prior use in the context of calibrating probabilistic models.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Temperature scaling is not best on every vision task

    Minor

    Table 1 reports lower ECE for histogram binning on Cars (1.74% versus 2.35%) and for BBQ on CIFAR-10 with ResNet 110 (0.54% versus 0.83%). Thus temperature scaling does not outperform all other methods on those vision tasks.

1 of 1

Citations

Cites

Flagged first, then by citations

Show all 37 references

Research works printed but not linked: 3

  1. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Cl´ ement. Torch7: A matlab-like environment for machine learning. In BigLearn Workshop, NIPS , 2011.
  2. Friedman, Jerome, Hastie, Trevor, and Tibshirani, Robert. The elements of statistical learning , volume 1. Springer series in statistics Springer, Berlin, 2001.
  3. Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS , 2011.

Cited by

Sorted by citations

See all 3,447

Related

See all

Metadata

Authors
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q.
Affiliations
Cornell University (US)
Corresponding author
Pleiss, Geoff
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
1321-1330
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1706.04599
arXiv
1706.04599
OpenAlex
W2626967530
Cited by
3,435
References
37
Citations by year
2026: 42 · 2025: 315 · 2024: 521 · 2023: 590 · 2022: 450 · 2021: 737 · 2020: 475 · 2019: 217 · 2018: 80 · 2017: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Funders
  • National Science Foundation (1618134, 1149882, 1526012)
  • Bill and Melinda Gates Foundation
  • Office of Naval Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
confidence calibration, artificial neural networks, temperature scaling, Platt scaling, probability estimates, network depth, weight decay, batch normalization, image classification, document classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review