CoChat
Sign Up
PreprintCited by 5,440Imbalanced Data Classification TechniquesData Mining and Machine Learning ApplicationsMachine Learning and Data Classification

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

Open access
Powers, David M. W.·2020·arXiv
DOI 10.48550/arxiv.2010.16061arXiv 2010.16061
Open this paper in CoChat — free accountOpen PDF

Abstract

Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.16061

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The test set mean F1-score (macro F1-score) is used as the main metric because the target datasets have substantial class imbalance and a metric resistant to class distribution bias is required.

    “We utilize the test set mean F1-score (i.e., macro F1-score) as the main metric to evaluate performance.This is motivated by the substantial class imbalance present in the target datasets (see Fig. 15 for reference), and thus we require a metric that is resistant to such bias in the class distribution [56].”

    From Assessing the State of Self-Supervised Human Activity Recognition Using Wearables (2022) · cites this paper as [56]

    The source makes the contrary point about F1: the Introduction (p. 1) says F-measure propagates underlying prevalence and bias, and “ROC and PN Analyses” (p. 5) shows F1 varies with prevalence and bias. Its multiclass simulation caption (p. 9) also describes harmonic-mean F1 as a biased traditional measure. The source does not assess the citing paper’s target datasets. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Traditional ML classification metrics propagate underlying biases, ignore the trustworthiness of negative instances, and do not take into account chance-level performance.

    “These traditional metrics, however, propagate the underlying biases, ignore the trustworthiness of negative instances and do not take into account the chance-level performance [79,81].”

    From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [79]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 19

  1. [1][1] Fraser, Alexander & Daniel Marcu (2007). Computational Linguistics 33(3):293-303. -
  2. [3][3] Lafferty, J., McCallum, A. & Pereira, F. (2001). 18th International Conference on Machine Learning , San Francisco (ICML), CA: Morgan Kaufmann, 282-289. -
  3. [4][4] Powers, David M. W. (2003), International Conference on Cognitive Science (ICCS). Accessed at http://david.wardpowers.info/BM/index.htm 2 March 2011.) -
  4. [5][5] Flach, PA. (2003). 20th International Conference on Machine Learning (ICML), Washington DC, 226-233. -
  5. [6][6] Fürnkranz Johannes & Peter A. Flach (2005). Machine Learning 58(1):39-77.
  6. [7][7] Lavrac, N., Flach, P., & Zupan, B. (1999). 9 th International Workshop on Inductive Logic Programming (ILP-99). Springer-Verlag, 174- 185.
  7. [8][8] Shanks, D. R. (1995). Quarterly Journal of Experimental Psychology , 48A, 257-279.
  8. [10][10] Bonett DG & RM Price, (2005). Journal of Educational and Behavioral Statistics 30(2) 213-225
  9. [11][11] Hutchinson TP. (1993). Research in Nursing & Health 16(4):313-316.
  10. [13][13] Carletta J. (1996). Computational Linguistics 22(2):249-254
  11. [15][15] Cohen J. (1968). Psychological Bulletin 70:21320.
  12. [16][16] Uebersax, J. S. (1987). Psychological Bulletin 101, 140 -146. Accessed expanded treatment at http:// ourworld.compuserve.com/homepages/ jsuebersax/agree.htm 28 February 2011.
  13. [20][20] Manning, Christopher D. and Hinrich Schütze (1999). Foundations of Statistical Natural Language Processing . MIT Press, Cambridge, MA. . Accessed 2 March 2011 as
  14. [23][23] Smith, PJ, Rae, DS, Manderscheid, RW and Silbergeld, S. (1981). Journal of the American Statistical Association 76:375,737-740.
  15. [25][25] Lewis, T. W. and D. M. W. Powers (2003). Journal of Research and Practice in Information Technology 35(1):41-64
  16. [26][26] Lewis, Trent W. (2003), Noise-robust Audio Visual Phoneme Recognition , PhD Thesis, School of Informatics and Engineering, Flinders University.
  17. [27][27] Fitzgibbon, Sean P., David M. W. Powers, Kenneth Pope, and C. Richard Clark (2007). Journal of Clinical Neurophysiology 24(3):232243
  18. [28][28] Fitzgibbon, Sean P (2007) A Machine Learning Approach to Brain-Computer Interfacing , PhD Thesis, School of Psychology, Flinders University.
  19. [30][30] Powers, David M. W. (2013). ADABOOK & MULTIBOOK: Adaptive Boosting with Chance Correction. ICINCO International Conference on Informatics in Control, Automation and Robotics, 349-359.

Cited by

Sorted by citations

Authors
Powers, D. M. W.
Affiliations
Flinders University (AU)
Corresponding author
Powers, David M. W.
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
2
Issue
1
Pages
37-63
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.16061
arXiv
2010.16061
OpenAlex
W46659105
Cited by
5,440
References
1
Citations by year
2026: 69 · 2025: 330 · 2024: 401 · 2023: 410 · 2022: 371 · 2021: 655 · 2020: 693 · 2019: 635 · 2018: 569 · 2017: 453 · 2016: 328 · 2015: 237 · 2014: 172 · 2013: 95 · 2012: 17
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Flinders Academic Commons (Flinders University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
precision, memory retrieval, F-measure, informedness, markedness, correlation, significance, base rates, multi-class classification, evaluation bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review