CoChat
Sign Up
Cited by 212Machine Learning and Data ClassificationImbalanced Data Classification TechniquesData Mining and Machine Learning Applications

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

100% Classification Accuracy Considered Harmful: The Normalized Information Transfer Factor Explains the Accuracy Paradox

Open access
Valverde-Albacete, Francisco J., Peláez-Moreno, Carmen·2014·PLoS ONE
DOI 10.1371/journal.pone.0084217PMC PMC3888391
Open this paper in CoChat — free accountOpen PDF

Abstract

The most widely spread measure of performance, accuracy, suffers from a paradox: predictive models with a given level of accuracy may have greater predictive power than models with higher accuracy. Despite optimizing classification error rate, high accuracy models may fail to capture crucial information transfer in the classification task. We present evidence of this behavior by means of a combinatorial analysis where every possible contingency matrix of 2, 3 and 4 classes classifiers are depicted on the entropy triangle, a more reliable information-theoretic tool for classification assessment. Motivated by this, we develop from first principles a measure of classification performance that takes into consideration the information learned by classifiers. We are then able to obtain the entropy-modulated accuracy (EMA), a pessimistic estimate of the expected accuracy with the influence of the input distribution factored out, and the normalized information transfer factor (NIT), a measure of how efficient is the transmission of information from the input to the output set of classes. The EMA is a more natural measure of classification performance than accuracy when the heuristic to maximize is the transfer of information through the classifier instead of classification error count. The NIT factor measures the effectiveness of the learning process in classifiers and also makes it harder for them to "cheat" using techniques like specialization, while also promoting the interpretability of results. Their use is demonstrated in a mind reading task competition that aims at decoding the identity of a video stimulus based on magnetoencephalography recordings. We show how the EMA and the NIT factor reject rankings based in accuracy, choosing more meaningful and interpretable classifiers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0084217

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Heatmap of the best classifiers of the MEG mind reading competition [23] according to accuracy (left) and the EMA and the NIT factor (right) criteria. Rows correspond to stimulus and columns to the decision or response. Darker hues correlate with higher joint probability . The heat map on the left reveals that the best classifier according to accuracy does not capture the fact that stimuli , and belong to a particular category whilst and belong to another. A B C
Figure 1 Heatmap of the best classifiers of the MEG mind reading competition [23] according to accuracy (left) and the EMA and the NIT factor (right) criteria. Rows correspond to stimulus and columns to the decision or response. Darker hues correlate with higher joint probability . The heat map on the left reveals that the best classifier according to accuracy does not capture the fact that stimuli , and belong to a particular category whilst and belong to another. A B CCC BY · PubMed Central
Figure 2 (Color online) Entropy decomposition for square matrices of (A) , (B) , and (C) (decimated), representing confusion matrices for a classification task at different accuracy levels as described by the right color bar. The interspersing of the plots representing matrices with different accuracies but similar entropies is evident at all levels for and but only for lower levels of accuracy for . This entails that accuracy is not a good criterion to judge the flow of information from the input labels to the output labels of a classifier (see text).
Figure 2 (Color online) Entropy decomposition for square matrices of (A) , (B) , and (C) (decimated), representing confusion matrices for a classification task at different accuracy levels as described by the right color bar. The interspersing of the plots representing matrices with different accuracies but similar entropies is evident at all levels for and but only for lower levels of accuracy for . This entails that accuracy is not a good criterion to judge the flow of information from the input labels to the output labels of a classifier (see text).CC BY · PubMed Central
Figure 3 (Color online) Entropy (above) and perplexity (below) decomposition chains for a joint distribution. Left, perplexity reduction in the input (learning) chain; right, perplexity increase in the output chain, related to classifier specialization. The colors refer to those of Fig. 5.(B). The ordering of the boxes is a convention to reveal the prior and posterior natures of the perplexities of class distributions.
Figure 3 (Color online) Entropy (above) and perplexity (below) decomposition chains for a joint distribution. Left, perplexity reduction in the input (learning) chain; right, perplexity increase in the output chain, related to classifier specialization. The colors refer to those of Fig. 5.(B). The ordering of the boxes is a convention to reveal the prior and posterior natures of the perplexities of class distributions.CC BY · PubMed Central
Figure 4 (Color online) Entropy triangle for the MEG mind Reading data ordered after accuracy (A) and a detail of the participants of higher accuracy (B). The ranking following accuracy is at odds with the EMA and the NIT factor ranking based in mutual information (height, right scale of triangle). The detail in (B) shows that participant , closely followed by should have been ranked first after this criterion.
Figure 4 (Color online) Entropy triangle for the MEG mind Reading data ordered after accuracy (A) and a detail of the participants of higher accuracy (B). The ranking following accuracy is at odds with the EMA and the NIT factor ranking based in mutual information (height, right scale of triangle). The detail in (B) shows that participant , closely followed by should have been ranked first after this criterion.CC BY · PubMed Central
Figure 5 (Color online) Extended information diagrams of entropies related to a bivariate distribution: (A) conventional diagram, and (B) split diagram. The bounding rectangle is the joint entropy of two uniform (thence independent) distributions and of the same cardinality as and . The expected mutual information appears twice in (A) and this makes the diagram split for each variable symmetrically in (B).
Figure 5 (Color online) Extended information diagrams of entropies related to a bivariate distribution: (A) conventional diagram, and (B) split diagram. The bounding rectangle is the joint entropy of two uniform (thence independent) distributions and of the same cardinality as and . The expected mutual information appears twice in (A) and this makes the diagram split for each variable symmetrically in (B).CC BY · PubMed Central
Figure 6 Schematic Entropy Triangle showing interpretable zones and extreme cases of classifiers. The annotations on the center of each side are meant to hold for that whole side.
Figure 6 Schematic Entropy Triangle showing interpretable zones and extreme cases of classifiers. The annotations on the center of each side are meant to hold for that whole side.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 29 references

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Valverde-Albacete, F. J., & Peláez-Moreno, C.
Affiliations
  • Universidad Nacional de Educación a Distancia (ES)
  • Universidad Carlos III de Madrid (ES)
Corresponding author
Valverde-Albacete, Francisco J.
ORCID
Venue
PLoS ONE
Publisher
Public Library of Science (PLoS)
Published
2014-01-10
Volume
9
Issue
1
Pages
e84217
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0084217
PMC
PMC3888391
OpenAlex
W2015363484
Cited by
212
References
29
Citations by year
2026: 13 · 2025: 12 · 2024: 27 · 2023: 14 · 2022: 35 · 2021: 38 · 2020: 20 · 2019: 21 · 2018: 12 · 2017: 8 · 2016: 8 · 2015: 1 · 2014: 2
FWCI
6.31
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (acceptedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • e-Archivo (Carlos III University of Madrid) (submittedVersion, cc-by)
  • Figshare (submittedVersion, cc-by)
Funders
European Commission (288024)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
Keywords
classification performance measures, contingency tables, information transmission, classifier evaluation, interpretability, theory of mind, magnetoencephalography, classification error rate
MeSH terms
Algorithms, Humans, Models, Theoretical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review