Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Validity, later known as reliability, is now often called calibration loss.

“Validity, later known as reliability (Murphy & Winkler, 1977), is now often called calibration loss (Kull & Flach, 2015).”

The decomposition of proper scoring rule into calibration loss and refinement loss is the sum of expected loss between S and C and expected loss between C and Y.

“The decomposition of proper scoring rule into calibration loss and refinement loss is then the sum of expected loss between S and C and expected loss between C and Y (Bröcker & Smith, 2007;Kull & Flach, 2015):”

For log-loss and Brier score, the divergence associated with a proper scoring rule corresponds to the Kullback-Leibler divergence and the mean squared difference, respectively.

“For log-loss and Brier score, has been shown to correspond to the Kullback-Leibler divergence and the mean squared difference, respectively (Kull & Flach, 2015).”

Binning approaches can reduce calibration loss at the cost of a higher grouping loss.

“In other words, binning approaches can reduce the calibration loss at the cost of a higher grouping loss (Kull & Flach, 2015).”

Assuming test data is representative of the whole population and there is a uniform distribution over test instances and zero probability elsewhere, empirical loss can be interpreted as a special case of expected loss and all decompositions derived for expected loss also apply to empirical loss.

“But assuming that test data is representative of the whole population and that there is a uniform distribution over the test instances and zero probability elsewhere, empirical loss can be interpreted as a special case of expected loss and all decompositions derived for expected loss also apply to empirical loss (Kull & Flach, 2015).”