The DOI matches its DataCite record; 49 of the 49 references it cites were found in a registry or an index; 2 of the 7 sources its claims cite were checked for support.
On Calibration of Modern Neural Networks
Open accessAbstract
Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Modern neural networks are often poorly calibrated: their predicted confidence exceeds their accuracy, unlike older networks. Temperature scaling, a single-parameter post-processing method, is surprisingly effective at fixing this across vision and NLP classification tasks.
Key findings
- Miscalibration is common in modern architectures: on CIFAR-100, a 110-layer ResNet has ECE 16.53% uncalibrated, while a 5-layer LeNet has ECE 4.85%; most datasets/models show ECE typically between 4 to 10%, with exceptions SVHN at 0.44% and Reuters at 0.85%.
- Depth, width, Batch Normalization, and weight decay affect calibration: increased model capacity and less weight decay are associated with worse ECE, and a 6-layer ConvNet with Batch Normalization has worse calibration even when accuracy improves slightly.
- Temperature scaling is often the most effective method: on CIFAR-100 ResNet 110 it reduces ECE from 16.53% to 1.26%, and on CIFAR-10 ResNet 110 from 4.6% to 0.83%; it outperforms vector and matrix scaling on vision tasks and performs comparably on NLP datasets.
- NLL can overfit after classification accuracy improves: on CIFAR-100, test error drops from 29% to 27% in the region where NLL overfits, suggesting networks gain accuracy at the expense of well-modeled probabilities.
Methods
- The paper evaluates post-processing calibration methods: histogram binning, isotonic regression, Bayesian Binning into Quantiles (BBQ), and Platt scaling variants including temperature scaling, vector scaling, and matrix scaling.
- Experiments cover 6 image datasets (Caltech-UCSD Birds, Stanford Cars, ImageNet 2012, CIFAR-10/100, SVHN) and 4 document/sentiment datasets (20 News, Reuters, SST binary, SST fine-grained), using ResNets, stochastic-depth ResNets, Wide ResNets, DenseNets, Deep Averaging Networks, and TreeLSTMs.
- Calibration is measured primarily by Expected Calibration Error (ECE) with M = 15 bins, with additional MCE, NLL, and test error tables; temperature scaling uses a single scalar T > 0 applied to logits and optimized by NLL on a validation set, leaving class predictions unchanged.
Limitations
- The paper states it cannot claim causality for the observed relationships between architecture/training trends and miscalibration, and says it remains future work to understand why these trends affect calibration while improving accuracy.
- Temperature scaling does not calibrate the Reuters dataset; the authors note this dataset is already well-calibrated (ECE ≤ 1%) and that measurements may be affected by dataset split or binning scheme.
- Matrix scaling performs poorly on datasets with hundreds of classes and fails to converge on 1000-class ImageNet, which the authors attribute to its number of parameters scaling quadratically with classes and overfitting a small validation set.
- The supplement states MCE is very sensitive to the binning scheme and less suited for small test sets.
- Evident from the text: all calibration methods require a hold-out validation set and assume training, validation, and test sets are drawn from the same distribution; evaluation is limited to the classification datasets and architectures listed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
29 Own result · 23 No basis given · 12 Own interpretation · 2 Sources not found · 1 Supported so far · 1 Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The only dataset that temperature scaling does not calibrate is the Reuters dataset.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The two notable exceptions are SVHN and Reuters, both of which experience ECE values below 1% .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“on most datasets, temperature scaling - a singleparameter variant of Platt Scaling - is surprisingly effective at calibrating predictions.”
- Also stated in: § 1. Introduction · p. 2 · plain
“we find that a single-parameter variant of Platt scaling (Platt et al., 1999) - which we refer to as temperature scaling - is often the most effective method at obtaining calibrated probabilities.”
What the sources say
- Not matchedcites Platt et al., 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The training, validation, and test sets are drawn from the same distribution.
- Also stated in: § 1. Introduction · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that this result holds regardless of the hyperparameters used on the Batch Normalization model (i.e. low or high learning rate, etc.).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the ECE metric grows substantially with model capacity.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, we observe a disconnect between NLL and accuracy , which may explain the miscalibration in Figure 2.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration.”
- Also stated in: § 3. Observing Miscalibration · p. 3 · hedged
“Though we cannot claim causality, we find that increased model capacity and lack of regularization are closely related to model miscalibration.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3. Observing Miscalibration · p. 3 · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On CIFAR-100, test error drops from 29% to 27% in the region where NLL overfits.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we do observe that models trained with Batch Normalization tend to be more miscalibrated.”
- Also stated in: § 3. Observing Miscalibration · p. 4 · plain
“a 6-layer ConvNet obtains worse calibration when Batch Normalization is applied, even though classification accuracy improves slightly.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3. Observing Miscalibration · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, overfitting to NLL is beneficial to classification accuracy.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 14
- It may state two separate claims as one. 22
- The quote only partly states this claim. 1
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Temperature scaling is commonly used in settings such as knowledge distillation (Hinton et al., 2015) and statistical mechanics (Jaynes, 1957).”
What the sources say
- Supportedcites Hinton et al., 2015 source
The cited Hinton et al. paper supports the knowledge-distillation portion of the sentence. Section 2 (p. 2) describes using a high temperature to produce soft targets and the same temperature to train the distilled model; the separate statistical-mechanics citation is not being assessed here. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites Jaynes, 1957 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent advances in deep learning have dramatically improved neural network accuracy (Simonyan & Zisserman, 2015; Srivastava et al., 2015; He et al., 2016; Huang et al., 2016; 2017).”
What the sources say
- Not matchedcites Simonyan & Zisserman, 2015
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Srivastava et al., 2015 source
The Introduction (p. 1) describes recent breakthroughs from deep neural networks and gives a substantial accuracy gain: top-5 ImageNet classification accuracy rose from about 84% to 95% in a few years with deeper architectures. This supports the citing sentence’s broad claim. (LLM, conf=0.92)
- Not matchedcites He et al., 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Temperature scaling outperforms all other methods on the vision tasks, and performs comparably to other methods on the NLP datasets.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Both of these datasets have very low error ( 1 . 98% and 2 . 97% , respectively); and therefore the ratio of ECE to error is comparable to other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, vector scaling recovers essentially the same solution as temperature scaling - the learned vector has nearly constant entries, and therefore is no different than a scalar transformation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“What is perhaps even more surprising is that temperature scaling outperforms the vector and matrix Platt scaling variants, which are strictly more general methods.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is worth noting that most datasets and models experience some degree of miscalibration, with ECE typically between 4 to 10% .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although increasing depth and width may reduce classification error, we observe that these increases negatively affect model calibration.”
- Also stated in: § 3. Observing Miscalibration · p. 3 · plain
“the ECE metric grows substantially with model capacity.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 3. Observing Miscalibration · p. 3 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Error with temperature scaling is exactly the same as uncalibrated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Hence ECE using M bins converges to the M -term Riemann-Stieltjes sum of E ˆ P [∣ ∣ ∣ P ( ˆ Y = Y | ˆ P = p ) -p ∣ ∣ ∣ ] .”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Temperature scaling not only lowers the NLL but also raises the entropy of the distribution (green line).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using a conjugate gradient solver, the optimal temperature can be found in 10 iterations, or a fraction of a second on most modern hardware.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Histogram binning and isotonic regression take an order of magnitude longer than temperature scaling, and BBQ takes roughly 3 orders of magnitude more time.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All methods scale linearly with the number of validation set samples.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Matrix scaling performs poorly on datasets with hundreds of classes (i.e. Birds, Cars, and CIFAR-100), and fails to converge on the 1000-class ImageNet dataset.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MCE seems very sensitive to the binning scheme and is less suited for small test sets.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each of the bins are well calibrated, which is remarkable given that all the probabilities were modified by only a single parameter.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To summarize, NLL roughly follows the trends of ECE.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, binning methods tend to change class predictions which hurts accuracy (see Section S3).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Temperature scaling is by far the fastest method, as it amounts to a onedimensional convex optimization problem.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Binning methods improve calibration on most datasets, but do not outperform temperature scaling.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The computational complexity of vector and matrix scaling are linear and quadratic respectively in the number of classes, reflecting the number of parameters in each method.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The observed disconnect between NLL and 0/1 loss suggests that these high capacity models are not necessarily immune from overfitting, but rather, overfitting manifests in probabilistic error rather than classification error.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increased model capacity will lower training NLL, and thus the model will be more (over)confident on average.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that reliability diagrams do not display the proportion of samples in a given bin, and thus cannot be used to estimate how many samples are calibrated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In all practical settings, achieving perfect calibration is impossible.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“It is a standard result (Friedman et al., 2001) that, in expectation, NLL is minimized if and only if ˆ π ( Y | X ) recovers the ground truth conditional distribution π ( Y | X ) .”
What the sources say
- Not matchedcites Friedman et al., 2001
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Basic probability tells us that acc( B m ) is an unbiased and consistent estimator of P ( ˆ Y = Y | ˆ P ∈ I m ) .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast, our framework, which does not augment the neural network model, returns a confidence score rather than returning a distribution of possible outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The model is equivalent to maximizing the entropy of the output probability distribution subject to certain constraints on the logits (see Section S2).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For perfectly calibrated classifiers, MCE and ECE both equal 0.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This results in a low-entropy softmax distribution over classes (blue line), which explains the model's overconfidence.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because this dataset is well-calibrated to begin with (ECE ≤ 1% ), there is not much room for improvement with any method”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The authors state this as an assumption.
The training, validation, and test sets are drawn from the same distribution.
“We assume that the training, validation, and test sets are drawn from the same distribution.”
The authors state this as an assumption.
There is no prior use of temperature scaling in the context of calibrating probabilistic models.
“To the best of our knowledge, we are not aware of any prior use in the context of calibrating probabilistic models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Confidence calibration was proposed by Guo et al. (2017), although not under that name.”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as Guo et al. (2017)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically, LLMs exhibited Expected Calibration Errors (ECE) ranging from 0.108 to 0.427, frequently expressing high confidence in incorrect answers [61].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Temperature scaling divides the logits vector by a learned scaling parameter, which is a simple yet effective approach to eliminate the over-confident problem of the output posteriors of neural networks [25].”
From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Traditional point predictions, even when accompanied by confidence scores from probabilistic models, often fail to quantify uncertainty reliably due to miscalibration [3], [4].”
From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [3]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“which was shown by Guo et al. (2017) to lead to well-calibrated predictions on the i.i.d. test set.”
From Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019) · cites this paper as Guo et al. (2017)
The cited source says what the claim attributes to it.
In the citing paper
“In addition, the method constructs its intervals from nonconformity scores based on the softmax scores [39], [40], which are known to suffer from calibration issues [41], [42].”
From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [41]
The cited source says what the claim attributes to it.
In the citing paper
“Deep neural networks are generally poorly calibrated and produce overconfident results.22”
From Appropriate trust in artificial intelligence for the optical diagnosis of colorectal polyps: the role of human/artificial intelligence interaction (2024) · cites this paper as [22]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, causal models have been used to estimate the impact of handover-parameter adjustments and other cellular-network configuration interventions, including transmission-power and cell-offset changes, on subsequent service and performance time series [121], [122], [123].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [122]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Language models can produce well-calibrated predictions for token probabilities on-distribution [Guo et al., 2017].”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Guo et al., 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5. Results · p. 7
“Temperature scaling outperforms all other methods on the vision tasks, and performs comparably to other methods on the NLP datasets.”
§ 4.2. Extension to Multiclass Models · p. 6
“| Cars | ResNet 50 | 4.3% | 1.74% | 4.29% | 1.84% | 2.35% | 2.37% | 10.5% | | CIFAR-10 | ResNet 110 | 4.6% | 0.58% | 0.81% | 0.54% | 0.83% | 0.88% | 1.0% |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 11×
“We train state-of-the-art convolutional networks: ResNets (He et al., 2016), ResNets with stochastic depth (SD) (Huang et al., 2016), Wide ResNets (Zagoruyko & Komodakis, 2016), and DenseNets (Huang et al., 2017).”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“ImageNet 2012 (Deng et al., 2009): Natural scene images from 1000 classes.”
BackgroundGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“This is visualized in Figure 1, which compares a 5-layer LeNet (left) (LeCun et al., 1998) with a 110-layer ResNet (right) (He et al.”
MethodDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · cited 4×
“We train state-of-the-art convolutional networks: ResNets (He et al., 2016), ResNets with stochastic depth (SD) (Huang et al., 2016), Wide ResNets (Zagoruyko & Komodakis, 2016), and DenseNets (Huang et al., 2017).”
BackgroundMethodInfluentialSelf-citationGirshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Wu, Y., & Vapnik, V. N. (1999) · Technometrics · doi:10.2307/1271368 · Cited by 27,236
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Research works printed but not linked: 3
- Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Cl´ ement. Torch7: A matlab-like environment for machine learning. In BigLearn Workshop, NIPS , 2011.
- Friedman, Jerome, Hastie, Trevor, and Tibshirani, Robert. The elements of statistical learning , volume 1. Springer series in statistics Springer, Berlin, 2001.
- Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS , 2011.
Cited by
Sorted by citations
Items 1,001–1,100 of 3,447
The loss surfaces of neural networks with general activation functions
Baskerville, N. P., Keating, J. P., Mezzadri, F., & Najnudel, J. (2021) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/abfa1e · Cited by 17
EEG Artifact Removal by Bayesian Deep Learning & ICA
Lee, S. S., Lee, K., & Kang, G. (2020) · Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc44109.2020.9175785 · Cited by 19
Towards image-based cancer cell lines authentication using deep neural networks
Mzurikwao, D., Khan, M. U., Samuel, O. W., Cinatl, J. J., Wass, M. N., Michaelis, M., Marcelli, G., & Ang, C. S. (2020) · Scientific Reports · doi:10.1038/s41598-020-76670-6 · Cited by 19
A deep analysis on high‐resolution dermoscopic image classification
Pollastri, F., Lara, M. P., Maroñas, J., Bolelli, F., Paredes, R., Ramos, D., & Grana, C. (2021) · IET Computer Vision · doi:10.1049/cvi2.12048 · Cited by 21
Regularizing Attention Networks for Anomaly Detection in Visual Question Answering
Lee, D., Cheon, Y., & Han, W.-S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16279 · Cited by 19
Reynolds, J., Gilmour, N. J., Baltazar, M. T., Reynolds, G., Windebank, S., & Maxwell, G. (2022) · Regulatory Toxicology and Pharmacology · doi:10.1016/j.yrtph.2022.105219 · Cited by 19
Wu, Y., He, K., Yan, Y., Gao, Q., Zeng, Z., Zheng, F., Zhao, L., Jiang, H., Wu, W., & Xu, W. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.307 · Cited by 19
Uncertainty Estimation for Heatmap-Based Landmark Localization
Schobs, L., Swift, A. J., & Lu, H. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3222730 · Cited by 20
Multi-task Active Learning for Pre-trained Transformer-based Models
Rotman, G., & Reichart, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00515 · Cited by 22
Deep Learning-Based Blind Multiple User Detection for Grant-Free SCMA and MUSA Systems
Sivalingam, T., Ali, S., Mahmood, N. H., et al. (2023) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2023.3283350 · Cited by 19
Ghosh, S., Belkhouja, T., Yan, Y., & Doppa, J. R. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25936 · Cited by 27
Calibrated Teacher for Sparsely Annotated Object Detection
Wang, H., Liu, L., Zhang, B., Zhang, J., Zhang, W., Gan, Z., Wang, Y., Wang, C., & Wang, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25349 · Cited by 24
Wang, Z., Luo, Y., Chen, Z., Wang, S., & Huang, Z. H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611808 · Cited by 19
Detection of Out-of-Distribution Samples Using Binary Neuron Activation Patterns
Olber, B., Radlak, K., Popowicz, A., Szczepankiewicz, M., & Chachuła, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00329 · Cited by 21
Trainable Projected Gradient Method for Robust Fine-Tuning
Tian, J., Dai, X., Ma, C., He, Z., Liu, Y., & Kira, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00757 · Cited by 29
A Primer on Bayesian Neural Networks: Review and Debates
Arbel, J., Pitas, K., Vladimirova, M., & Fortuin, V. (2026) · Statistical Science · doi:10.1214/24-sts969 · Cited by 78
Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction
Park, S., Cohen, K. M., & Simeone, O. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327300 · Cited by 19
Ma, C., & Wolfinger, R. D. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad399 · Cited by 21
Selectively Answering Ambiguous Questions
Cole, J., Zhang, M., Gillick, D., Eisenschlos, J., Dhingra, B., & Eisenstein, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.35 · Cited by 19
Mahapatra, D., Bozorgtabar, B., Ge, Z., & Reyes, M. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103075 · Cited by 20
Fixing Overconfidence in Dynamic Neural Networks
Meronen, L., Trapp, M., Pilzer, A., Yang, L., & Solin, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00266 · Cited by 29
VSR-Net: Vessel-Like Structure Rehabilitation Network With Graph Clustering
Ye, H., Zhang, X., Hu, Y., Fu, H., & Liu, J. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3526061 · Cited by 19
A framework for explainable deep neural models using external knowledge graphs
Daniels, Z., Frank, L., Menart, C., Raymer, M. L., & Hitzler, P. (2020) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.2558083 · Cited by 19
Maier, A., Lorch, B., & Rieß, C. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191121 · Cited by 19
Deep Neural Networks Classification via Binary Error-Detecting Output Codes
Klimo, M., Lukáč, P., & Tarábek, P. (2021) · Applied Sciences · doi:10.3390/app11083563 · Cited by 19
The Group Loss++: A Deeper Look Into Group Loss for Deep Metric Learning
Elezi, I., Seidenschwarz, J., Wagner, L., Vascon, S., Torcinovich, A., Pelillo, M., & Leal-Taixé, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3163846 · Cited by 19
Calibrate: Interactive Analysis of Probabilistic Model Output
Xenopoulos, P., Rulff, J., Nonato, L. G., Barr, B., & Silva, C. T. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209489 · Cited by 21
Confidence Scoring Using Whitebox Meta-models with Linear Classifier ProbesPreprint
Chen, T., Navrátil, J., Iyengar, V., & Shanmugam, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.05396 · Cited by 21
xGEMs: Generating Examplars to Explain Black-Box ModelsPreprint
Joshi, S. D., Koyejo, S., Kim, B., & Ghosh, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08867 · Cited by 19
Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data StreamsPreprint
Kachuee, M., Goldstein, O., Kärkkäinen, K., Darabi, S., & Sarrafzadeh, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00243 · Cited by 32
On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty EstimationPreprint
Brosse, N., Riquelme, C., Martin, A. O., Gelly, S., & Moulines, É. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08049 · Cited by 35
scNym: Semi-supervised adversarial neural networks for single cell classificationPreprint
Kimmel, J. C., & Kelley, D. R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.04.132324 · Cited by 19
One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR PredictionPreprint
Sheng, X.-R., Zhao, L., Zhou, G., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11427 · Cited by 19
Uncertainty-Aware Machine Translation Evaluation
Glushkova, T., Zerva, C., Rei, R., & Martins, A. F. T. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.330 · Cited by 17
A Bayesian Neural Net to Segment Images with Uncertainty Estimates and Good Calibration
Jena, R., & Awate, S. P. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20351-1_1 · Cited by 32
Large Language Models Are Unreliable for Cyber Threat Intelligence
Mezzi, E., Massacci, F., & Tuma, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00627-1_17 · Cited by 37
Distance-Based Learning from Errors for Confidence Calibration
Chen, X., Arık, S. Ö., Zhang, Z., & Pfister, T. (2020) · International Conference on Learning Representations · Cited by 19
An Imitation Game for Learning Semantic Parsers from User Interaction
Yao, Z., Tang, Y., Yih, W.-T., Sun, H., & Su, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.559 · Cited by 21
Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
An, B., Lyu, J., Wang, Z., Li, C., Hu, C., Tan, F., Zhang, R., Hu, Y., & Chen, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.17 · Cited by 26
González, A. V., Bansal, G., Fan, A., Mehdad, Y., Jia, R., & Iyer, S. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.95 · Cited by 20
Well-Classified Examples Are Underestimated in Classification with Deep Neural Networks
Zhao, G., Yang, W., Ren, X., Li, L., Wu, Y., & Sun, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20904 · Cited by 20
Ravindran, P., Owens, F. C., Wade, A. C., Shmulsky, R., & Wiedenhoeft, A. C. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2021.758455 · Cited by 18
Das, S., Wang, Y., Gong, J., Ding, L., Munchak, S. J., Wang, C., Wu, D. L., Liao, L., Olson, W. S., & Barahona, D. (2022) · Remote Sensing · doi:10.3390/rs14153631 · Cited by 18
Space Applications of a Trusted AI Framework: Experiences and Lessons Learned
Mandrake, L., Doran, G. B., Goel, A., Ono, H., Amini, R., Feather, M. S., Fesq, L. M., Slingerland, P. C., Perry, L., Bycroft, B. P., et al. (2022) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero53065.2022.9843322 · Cited by 18
Graph Node Based Interpretability Guided Sample Selection for Active Learning
Mahapatra, D., Poellinger, A., & Reyes, M. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3215017 · Cited by 18
Bayesian Active Meta-Learning for Reliable and Efficient AI-Based Demodulation
Cohen, K., Park, S., Simeone, O., & Shamai, S. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3220035 · Cited by 16
Machine learning models for predicting severe COVID-19 outcomes in hospitals
Wendland, P., Schmitt, V., Zimmermann, J., Häger, L., Göpel, S., Schenkel-Häger, C., & Kschischo, M. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101188 · Cited by 18
Image-Consistent Detection of Road Anomalies as Unpredictable Patches
Vojir, T., & Matas, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00545 · Cited by 20
Gruich, C., Madhavan, V., Wang, Z., & Goldsmith, B. R. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/accace · Cited by 18
Pacheco, R. G., Couto, R. S., & Simeone, O. (2023) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2023.103679 · Cited by 18
Probabilistic Neural Data Fusion for Learning from an Arbitrary Number of Multi-Fidelity Data Sets
Mora, C., Eweis-Labolle, J. T., Johnson, T. B., Gadde, L., & Bostanabad, R. (2023) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2023.116207 · Cited by 20
BotPercent: Estimating Bot Populations in Twitter Communities
Tan, Z., Feng, S., Sclar, M., Wan, H., Luo, M., Choi, Y., & Tsvetkov, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.954 · Cited by 23
Unleashing the Potential of Spiking Neural Networks with Dynamic Confidence
Li, C., Jones, E. G., & Furber, S. B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01228 · Cited by 29
Conformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertainty
Mossina, L., Dalmau, J., & Andéol, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00361 · Cited by 18
Deep Learning in Breast Cancer Screening
Harvey, H., Heindl, A., Khara, G. S., Korkinof, D., O’Neill, M., Yearsley, J. E., Karpati, E., Rijken, T. M., Kecskemethy, P. D., & Forrai, G. (2019) · Springer eBooks · doi:10.1007/978-3-319-94878-2_14 · Cited by 18
A General Framework for Ensemble Distribution Distillation
Lindqvist, J., Olmin, A., Lindsten, F., & Svensson, L. (2020) · Machine learning for signal processing ... · doi:10.1109/mlsp49062.2020.9231703 · Cited by 18
Bertels, J., Robben, D., Vandermeulen, D., & Suetens, P. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101833 · Cited by 19
A Confidence-Calibrated MOBA Game Winner Predictor
Kim, D., Lee, C.-W., & Chung, K. (2020) · IEEE Conference on Games (CoG) · doi:10.1109/cog47356.2020.9231878 · Cited by 18
DiscFace: Minimum Discrepancy Learning for Deep Face Recognition
Kim, I., Han, S., Park, S.-J., Baek, J.-W., Shin, J., Han, J., & Choi, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69541-5_22 · Cited by 18
Calibrated Adversarial Refinement for Stochastic Semantic Segmentation
Kassapis, E., Dikov, G., Gupta, D. K., & Nugteren, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00697 · Cited by 18
A Bayesian Approach to In-Game Win Probability in Soccer
Robberechts, P., Haaren, J. V., & Davis, J. J. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467194 · Cited by 18
Conformance Constraint Discovery
Fariha, A., Tiwari, A., Radhakrishna, A., Gulwani, S., & Meliou, A. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3452795 · Cited by 18
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 23
Calibrating Uncertainties in Object Localization TaskPreprint
Phan, B., Salay, R., Czarnecki, K., Abdelzad, V., Denouden, T. A., & Vernekar, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.11210 · Cited by 23
Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine SimilarityPreprint
Techapanurak, E., Suganuma, M., & Okatani, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10628 · Cited by 23
Probability Calibration for Knowledge Graph Embedding ModelsPreprint
Tabacof, P., & Costabello, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10000 · Cited by 22
Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data BiasPreprint
Picard, R. W., & Ghandeharioun, A. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 19
Wat zei je? Detecting Out-of-Distribution Translations with Variational TransformersPreprint
Xiao, T. Z., Gomez, A. N., & Gal, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08344 · Cited by 33
Distribution-free binary classification: prediction sets, confidence intervals and calibrationPreprint
Gupta, C., Podkopaev, A., & Ramdas, A. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2006.10564 · Cited by 106
Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric UncertaintyPreprint
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H. S., & Gal, Y. (2021) · arXiv (Cornell University) · Cited by 18
Learning Optimal Conformal ClassifiersPreprint
Stutz, D., Krishnamurthy, Dvijotham, Cemgil, A. T., & Arnaud Doucet, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09192 · Cited by 70
EvidenceCap: Towards trustworthy medical image segmentation via evidential identity capPreprint
Zou, K., Yuan, X., Shen, X., Chen, Y., Wang, M., Goh, R. S. M., Liu, Y., & Fu, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2558155/v1 · Cited by 18
SURE: SUrvey REcipes for Building Reliable and Robust Deep Networks
Li, Y., Chen, Y., Yu, X., Chen, D., & Shen, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01657 · Cited by 18
Decoupling Inherent Risk and Early Cancer Signs in Image-Based Breast Cancer Risk Models
Liu, Y., Azizpour, H., Strand, F., & Smith, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59725-2_23 · Cited by 17
IST-Unbabel Participation in the WMT20 Quality Estimation Shared Task
Moura, J., Vera, M., van Stigt, D., et al. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.119 · Cited by 17
Confidence Calibration for Deep Renal Biopsy Immunofluorescence Image Classification
Pollastri, F., Maroñas, J., Bolelli, F., Ligabue, G., Paredes, R., Magistroni, R., & Grana, C. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412685 · Cited by 17
Can Explanations Be Useful for Calibrating Black Box Models?
Ye, X., & Durrett, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.429 · Cited by 23
Towards Textual Out-of-Domain Detection Without In-Domain Labels
Jin, D., Gao, S., Kim, S., Liu, Y., & Hakkani-Tür, D. Z. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3162081 · Cited by 18
Hyperbolic Uncertainty Aware Semantic Segmentation
Chen, B., Peng, W., Cao, X., & Röning, J. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3312290 · Cited by 24
Meta-Learning for Fast and Privacy-Preserving Source Knowledge Transfer of EEG-Based BCIs
Li, S., Wu, H., Ding, L., & Wu, D. (2022) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2022.3199622 · Cited by 17
Evaluation of at-home physiotherapy
Boyer, P., Burns, D., & Whyne, C. M. (2023) · Bone and Joint Research · doi:10.1302/2046-3758.123.bjr-2022-0126.r1 · Cited by 17
Understanding the Challenges When 3D Semantic Segmentation Faces Class Imbalanced and OOD Data
Pan, Y., Xie, F., & Zhao, H. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3256442 · Cited by 17
Margin-aware rectified augmentation for long-tailed recognition
Xiang, L., Han, J., & Ding, G. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109608 · Cited by 16
Probabilistic Prognosis of Wind Turbine Faults With Feature Selection and Confidence Calibration
Xu, J. B., Jiang, X., Liao, S., Ke, D., Sun, Y., Yao, L., & Mao, B. (2023) · IEEE Transactions on Sustainable Energy · doi:10.1109/tste.2023.3272317 · Cited by 17
Self-citationRevisiting Softmax for Uncertainty Approximation in Text Classification
Holm, A. N., Wright, D., & Augenstein, I. (2023) · Information · doi:10.3390/info14070420 · Cited by 17
Mehrtens, H. A., Kurz, A., Bucher, T.-C., & Brinker, T. J. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102914 · Cited by 27
Uncertainty-Aware Unlikelihood Learning Improves Generative Aspect Sentiment Quad Prediction
Hu, M., Bai, Y., Wu, Y., Zhang, Z., Zhang, L., Gao, H., Zhao, S., & Huang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.851 · Cited by 19
Class Adaptive Network Calibration
Liu, B., Rony, J., Galdrán, A., Dolz, J., & Ayed, I. B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01542 · Cited by 18
Model Ensemble With Dropout for Uncertainty Estimation in Sea Ice Segmentation Using Sentinel-1 SAR
de Lima, R. P., & Karimzadeh, M. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3331276 · Cited by 17
Improving Seq2Seq Grammatical Error Correction via Decoding Interventions
Zhou, H., Liu, Y., Li, Z., Zhang, M., Zhang, B., Li, C., Zhang, J., & Huang, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.495 · Cited by 21
Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers
Tam, W., Liu, X., Ji, K., Xue, L., Liu, J., Li, T., Dong, Y., & Tang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.874 · Cited by 33
Far Away in the Deep Space: Dense Nearest-Neighbor-Based Out-of-Distribution Detection
Galesso, S., Argus, M. J., & Brox, T. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00482 · Cited by 19
Barrado, M. R., Villaverde, T., Perez, M. F., Sanmartín, I., & Riina, R. (2024) · Annals of Botany · doi:10.1093/aob/mcae001 · Cited by 17
DiffGuard: Semantic Mismatch-Guided Out-of-Distribution Detection using Pre-trained Diffusion Models
Gao, R., Zhao, C., Hong, L., & Xu, Q. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00152 · Cited by 18
Improving Open Intent Detection via Triplet-Contrastive Learning and Adaptive Boundary
Chen, G., Xu, Q., Zhan, C., Wang, F. L., Liu, K., Liu, H., & Hao, T. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3363896 · Cited by 18
Li, G., Wang, G., Guo, Z., Liu, Q., Luo, X., Yuan, B., Li, M., & Yang, L. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3367665 · Cited by 20
Controller Synthesis for Autonomous Systems With Deep-Learning Perception Components
Călinescu, R., Imrie, C., Mangal, R., Rodrigues, G. N., Păsăreanu, C. S., Santana, M. A., & Vázquez, G. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3385378 · Cited by 23
One Fits Many: Class Confusion Loss for Versatile Domain Adaptation
Jin, Y. L., Cao, Z., Wang, X., Wang, J., & Long, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3392565 · Cited by 18
A Cooperative Training Framework for Underwater Object Detection on a Clearer View
Chen, G., Mao, Z., Tu, Q., et al. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3440386 · Cited by 17
Feng, Y., Zheng, C., Chen, J., Pan, T., Xie, J., He, S., & Wang, H. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.112057 · Cited by 21
Metadata
- Authors
- Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q.
- Affiliations
- Cornell University (US)
- Corresponding author
- Pleiss, Geoff
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Pages
- 1321-1330
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1706.04599
- OpenAlex
- W2626967530
- Cited by
- 3,435
- References
- 37
- Citations by year
- 2026: 42 · 2025: 315 · 2024: 521 · 2023: 590 · 2022: 450 · 2021: 737 · 2020: 475 · 2019: 217 · 2018: 80 · 2017: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- National Science Foundation (1618134, 1149882, 1526012)
- Bill and Melinda Gates Foundation
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- confidence calibration, artificial neural networks, temperature scaling, Platt scaling, probability estimates, network depth, weight decay, batch normalization, image classification, document classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
