CoChat
Sign Up
Cited by 99

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Deconstructing Cross-Entropy for Probabilistic Binary Classifiers

Ramos, Daniel, Franco-Pedroso, Javier, Lozano-Diez, Alicia, Gonzalez-Rodriguez, Joaquin·2018·Entropy
DOI 10.3390/e20030208PMC PMC7512723
Open this paper in CoChat — free account

Abstract

In this work, we analyze the cross-entropy function, widely used in classifiers both as a performance measure and as an optimization objective. We contextualize cross-entropy in the light of Bayesian decision theory, the formal probabilistic framework for making decisions, and we thoroughly analyze its motivation, meaning and interpretation from an information-theoretical point of view. In this sense, this article presents several contributions: First, we explicitly analyze the contribution to cross-entropy of (i) prior knowledge; and (ii) the value of the features in the form of a likelihood ratio. Second, we introduce a decomposition of cross-entropy into two components: discrimination and calibration. This decomposition enables the measurement of different performance aspects of a classifier in a more precise way; and justifies previously reported strategies to obtain reliable probabilities by means of the calibration of the output of a discriminating classifier. Third, we give different information-theoretical interpretations of cross-entropy, which can be useful in different application scenarios, and which are related to the concept of reference probabilities. Fourth, we present an analysis tool, the Empirical Cross-Entropy (ECE) plot, a compact representation of cross-entropy and its aforementioned decomposition. We show the power of ECE plots, as compared to other classical performance representations, in two diverse experimental examples: a speaker verification system, and a forensic case where some glass findings are present.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/e20030208

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 9 references

Research works printed but not linked: 34

  1. Murphy, K.P. (2012). Machine Learning: A Probabilistic Perspective, MIT Press.
  2. Smola, A.J., Bartlett, P., Sholkopf, B., and Shchuurmans, D. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, MIT Press. [Book ed.]. Chapter 10.
  3. Zadrozny, B., and Elkan, C. (2002, January 23–26). Transforming classifier scores into accurate multiclass probability estimates. In Proceeding of the Eight International Conference on Knowledge Discovery and Data Mining (KDD’02), Edmonton, AB, Canada. · DOI
  4. Cohen, I., and Goldszmidt, M. (2004). Properties and benefits of calibrated classifiers. Lecture Notes in Computer Science, Springer. Knowledge Discovery in Databases: PKDD 2004. · DOI
  5. Niculescu-Mizil, A., and Caruana, R. (2005, January 7–11). Predicting Good Probabilities With Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany. · DOI · Paper page
  6. Kittler, 1998, On combining classifiers, IEEE Trans. Pattern Anal. Mach. Intell., 20, 226, 10.1109/34.667881 · DOI
  7. Koller, D., and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques—Adaptive Computation and Machine Learning, MIT Press.
  8. Sim, 2001, Clinical Decision Support Systems for the Practice of Evidence-based Medicine, J. Am. Med. Inform. Assoc., 8, 527, 10.1136/jamia.2001.0080527 · DOI · Paper page
  9. Gigerenzer, 1991, Probabilistic Mental Models: A Brunswikian Theory of Confidence, Psychol. Rev., 98, 506, 10.1037/0033-295X.98.4.506 · DOI · Paper page
  10. Müller, C. (2007). An introduction to application-independent evaluation of speaker recognition systems. Speaker Classification, Springer. Lecture Notes in Computer Science/Artificial Intelligence.
  11. 2006, Application Independent Evaluation of Speaker Detection, Comput. Speech Lang., 20, 230, 10.1016/j.csl.2005.08.001 · DOI
  12. Tistarelli, M., and Champod, C. (2017). From Bometric Scores to Forensic Likelihood Ratios. Handbook of Biometrics for Forensic Science, Book ed., Springer. Chapter 14. · DOI
  13. Ramos, 2013, Reliable support: measuring calibration of likelihood ratios, Forensic Sci. Int., 230, 156, 10.1016/j.forsciint.2013.04.014 · DOI · Paper page
  14. Berger, 2011, Expressing evaluative opinions: A position statement, Sci. Justice, 51, 1, 10.1016/j.scijus.2011.01.002 · DOI
  15. Savage, 1971, The elicitation of personal probabilities and expectations, J. Am. Stat. Assoc., 66, 783, 10.1080/01621459.1971.10482346 · DOI
  16. Gneiting, 2007, Strictly Proper Scoring Rules, Prediction and Estimation, J. Am. Stat. Assoc., 102, 359, 10.1198/016214506000001437 · DOI · Paper page
  17. Richard, 1991, Neural network classifiers estimate Bayesian a posteriori probabilities, Neural Comput., 3, 461, 10.1162/neco.1991.3.4.461 · DOI
  18. Ramos, 2013, Information-Theoretical Assessment of the Performance of Likelihood Ratio Models, J. Forensic Sci., 58, 1503, 10.1111/1556-4029.12233 · DOI
  19. Kinnunen, 2010, An overview of text-independent speaker recognition: From features to supervectors, Speech Commun., 52, 12, 10.1016/j.specom.2009.08.009 · DOI
  20. Shannon, 1948, A Mathematical Theory of Communication, Bell Syst. Tech. J., 27, 379, 10.1002/j.1538-7305.1948.tb01338.x · DOI · Paper page
  21. Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley Interscience. [2nd ed.].
  22. Brümmer, N. (2010). Measuring, Refining and Calibrating Speaker and Language Information Extracted from Speech. [Ph.D. Thesis, School of Electrical Engineering, University of Stellenbosch]. Available online: http://sites.google.com/site/nikobrummer/.
  23. Dehak, 2010, Front-End Factor Analysis for Speaker Verification, IEEE Trans. Audio Speech Lang. Process., 19, 788, 10.1109/TASL.2010.2064307 · DOI · Paper page
  24. Kenny, P. (2010). Bayesian speaker verification with heavy-tailed priors. Odyssey: The Speaker and Language Recognition Workshop, International Speech Communication Association.
  25. Burget, 2007, Fusion of heterogeneous speaker recognition systems in the STBU submission for the NIST speaker recognition evaluation 2006, IEEE Trans. Audio Speech Signal Process., 15, 2072, 10.1109/TASL.2007.902870 · DOI
  26. Martin, A., and Greenberg, C. (2010, January 26–30). The NIST 2010 speaker recognition evaluation. Proceedings of the Interspeech 2010, Makuhari, Chiba, Japan. · DOI
  27. Martin, A., and Greenberg, C. (2009, January 6–10). NIST 2008 Speaker Recognition Evaluation: Performance Across Telephone and Room Microphone Channels. Proceedings of the Interspeech 2009, Brighton, UK. · DOI
  28. Aitken, 2004, Evaluation of trace evidence in the form of multivariate data, Appl. Stat., 53, 109
  29. Franco-Pedroso, J., Ramos, D., and Gonzalez-Rodriguez, J. (2016). Gaussian Mixture Models of Between-Source Variation for Likelihood Ratio Computation from Multivariate Data. PLoS ONE, 11. · DOI
  30. Thompson, 2015, Lay understanding of forensic statistics: Evaluation of random match probabilities, likelihood ratios, and verbal equivalents, Law Hum. Behav., 39, 332, 10.1037/lhb0000134 · DOI
  31. Wei, 2010, A novel measure for evaluating classifiers, Expert Syst. Appl., 37, 3799, 10.1016/j.eswa.2009.11.040 · DOI
  32. Corzo, R., Hoffman, T., Weis, P., Franco-Pedroso, J., Ramos, D., and Almirall, J. (2018). The Use of LA-ICP-MS Databases to Estimate Likelihood Ratios for the Forensic Analysis of Glass Evidence. Talanta, in press. · DOI
  33. Rose, 2007, Emulating DNA: Rigorous Quantification of Evidential Weight in Transparent and Testable Forensic Speaker Recognition, IEEE Trans. Audio Speech Lang. Process., 15, 2072
  34. Morrison, 2013, Tutorial on logistic-regression calibration and fusion: Converting a score to a likelihood ratio, Aust. J. Forensic Sci., 45, 173, 10.1080/00450618.2012.733025 · DOI

Cited by

Sorted by citations

See all 99

Metadata

Authors
Ramos, D., Franco-Pedroso, J., Lozano-Diez, A., & Gonzalez-Rodriguez, J.
Affiliations
AuDIaS-Audio, Data Intelligence and Speech, Escuela Politecnica Superior, Universidad Autonoma de Madrid, Calle Francisco Tomas y Valiente 11, 28049 Madrid, Spain
ORCID
Venue
Entropy
Publisher
MDPI AG
Published
2018-03-20
Volume
20
Issue
3
Pages
208
ISSN
1099-4300
DOI
10.3390/e20030208
PMC
PMC7512723
Cited by
99
References
9
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)
Funders
Spanish Ministry of Economy and Competitiveness (TEC2015-68172-C2-1-P)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review