A review of the F-measure: its history, properties, criticism, and alternatives
Open accessAbstract
Methods to classify objects into two or more classes are at the core of various disciplines. When a set of objects with their true classes is available, a supervised classifier can be trained and employed to decide if, for example, a new patient has cancer or not. The choice of performance measure is critical in deciding which supervised method to use in any particular classification problem. Different measures can lead to very different choices, so the measure should match the objectives. Many performance measures have been developed, and one of them is the F-measure, the harmonic mean of precision and recall. Originally proposed in information retrieval, the F-measure has gained increasing interest in the context of classification. However, the rationale underlying this measure appears weak, and unlike other measures, it does not have a representational meaning. The use of the harmonic mean also has little theoretical justification. The F-measure also stresses one class, which seems inappropriate for general classification problems. We provide a history of the F-measure and its use in computational disciplines, describe its properties, and discuss criticism about the F-Measure. We conclude with alternatives to the F-measure, and recommendations of how to use it effectively.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
F1 has no intuitive interpretation.
“(2) F1 has no intuitive interpretation”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [78]
F1 and MCC were introduced to address the challenges caused by class imbalance, where classification accuracy can be inflated by classifying all patients as low risk when the event is rare.
“F1 and MCC were introduced to address the challenges caused by class imbalance, where classification accuracy can be inflated by classifying all patients as low risk when the event is rare.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [78]
The absolute value of F1 changes by simply switching the outcome labels (when 1 becomes 0 and 0 becomes 1).
“(3) the absolute value of F1 changes by simply switching the outcome labels (ie, when 1 becomes 0 and 0 becomes 1).”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [78]
These issues hold for the more general F beta class of measures, of which F1 is a special case.
“These issues hold for the more general F beta class of measures, of which F1 is a special case.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [78]
F1 ignores true negatives.
“(1) F1 ignores true negatives”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [78]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 11,966 · see citing works
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288 · see citing works
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · see citing works
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515 · see citing works
Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00075 · Cited by 6,612 · see citing works
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 1× · see citing works
“Various reviews which look at different aspects of classification performance measures have been written, including [11, 15, 19, 23, 27, 50, 57], and others.”
BackgroundIntroduction to modern information retrieval
Dillon, M. (1983) · Information Processing & Management · doi:10.1016/0306-4573(83)90062-6 · Cited by 5,980 · see citing works
Frequency-tuned salient region detection
Achanta, R., Hemami, S. S., Estrada, F., & Susstrunk, S. E. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206596 · Cited by 4,368 · see citing works
Nouvelles recherches sur la distribution florale
Jaccard, P. (1908) · E-Periodica · doi:10.5169/seals-268384 · Cited by 3,352 · see citing works
2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,846 · see citing works
Deep Learning--based Text Classification
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad-Khasmakhi, N., Asgari-Chenaghlu, M., & Gao, J. (2021) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,967 · see citing works
Detecting text in natural scenes with stroke width transform
Epshtein, B., Ofek, E., & Wexler, Y. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540041 · Cited by 1,526 · see citing works
A Method for Comparing Two Hierarchical Clusterings
Fowlkes, E. B., & Mallows, C. L. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10478008 · Cited by 1,515 · see citing works
Kosala, R., & Blockeel, H. (2000) · ACM SIGKDD Explorations Newsletter · doi:10.1145/360402.360406 · Cited by 1,465 · see citing works
Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases
Janowczyk, A. R., & Madabhushi, A. (2016) · Journal of Pathology Informatics · doi:10.4103/2153-3539.186902 · Cited by 1,355 · see citing works
A Survey of Predictive Modeling on Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · ACM Computing Surveys · doi:10.1145/2907070 · Cited by 1,129 · see citing works
Detection theory: A user's guide, 2nd ed.
Macmillan, N. A., & Creelman, C. D. (2005) · Cited by 1,117 · see citing works
Measuring classifier performance: a coherent alternative to the area under the ROC curve
Hand, D. J. (2009) · Machine Learning · doi:10.1007/s10994-009-5119-5 · Cited by 1,037 · see citing works
Self-citationAn experimental comparison of performance measures for classification
Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.010 · Cited by 889 · see citing works
A support vector method for multivariate performance measures
Joachims, T. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102399 · Cited by 795 · see citing works
Training and assessing classification rules with imbalanced data
Menardi, G., & Torelli, N. (2012) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-012-0295-5 · Cited by 769 · see citing works
Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000060 · Cited by 723 · see citing works
Self-citationEvaluation of background subtraction techniques for video surveillance
Brutzer, S., Höferlin, B., & Heidemann, G. (2011) · doi:10.1109/cvpr.2011.5995508 · Cited by 630 · see citing works
Ensemble of keyword extraction methods and classifiers in text classification
Onan, A., Korukoğlu, S., & Bulut, H. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.03.045 · Cited by 630 · see citing works
Optimal Thresholding of Classifiers to Maximize F1 Measure
Lipton, Z. C., Elkan, C., & Naryanaswamy, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44851-9_15 · Cited by 597 · see citing works
Chinchor, N. A. (1992) · Message Understanding Conference (MUC) · doi:10.3115/1072064.1072067 · Cited by 575 · see citing works
Construction and Assessment of Classification Rules.
Ringrose, T. J., & Hand, D. J. (1997) · Biometrics · doi:10.2307/2533581 · Cited by 490 · see citing works
Self-citationEvaluating and optimizing autonomous text classification systems
Lewis, D. D. (1995) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/215206.215366 · Cited by 316 · see citing works
AutoML to Date and Beyond: Challenges and Opportunities
Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., & Veeramachaneni, K. K. (2021) · ACM Computing Surveys · doi:10.1145/3470918 · Cited by 294 · see citing works
Statistics and the Theory of Measurement
Hand, D. J. (1996) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983326 · Cited by 264 · see citing works
Self-citationA note on using the F-measure for evaluating record linkage algorithms
Hand, D., & Christen, P. (2018) · Statistics and Computing · doi:10.1007/s11222-017-9746-6 · Cited by 247 · see citing works
Self-citationF*: an interpretable transformation of the F-measure
Hand, D. J., Christen, P., & Kirielle, N. (2021) · Machine Learning · doi:10.1007/s10994-021-05964-1 · Cited by 224 · see citing works
Self-citationAdaptive least squares support vector machines filter for hand tremor canceling in microsurgery
Liu, Z., Wu, Q., Zhang, Y., & Chen, C. L. P. (2011) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-011-0012-5 · Cited by 214 · see citing works
Buja, A., Stuetzle, W., & Shen, Y. (2005) · Cited by 202 · see citing works
Assessing the Performance of Classification Methods
Hand, D. J. (2012) · International Statistical Review · doi:10.1111/j.1751-5823.2012.00183.x · Cited by 177 · see citing works
Self-citationMeasurement Theory and Practice: The World Through Quantification
McCullough, B. D. (2005) · Journal of the American Statistical Association · doi:10.1198/jasa.2005.s56 · Cited by 134 · see citing works
(Almost) all of entity resolution
Binette, O., & Steorts, R. C. (2022) · Science Advances · doi:10.1126/sciadv.abi8021 · Cited by 107 · see citing works
A note on comparing classifiers
Duin, R. P. W. (1996) · Pattern Recognition Letters · doi:10.1016/0167-8655(95)00113-1 · Cited by 88 · see citing works
Learning from Imbalanced Data: Evaluation Matters
Raeder, T., Forman, G., & Chawla, N. V. (2011) · Intelligent systems reference library · doi:10.1007/978-3-642-23166-7_12 · Cited by 77 · see citing works
Christen, P., Ranbaduge, T., & Schnell, R. (2020) · Springer eBooks · doi:10.1007/978-3-030-59706-1 · Cited by 74 · see citing works
Self-citationA Pragmatic Perspective of Measurement
Irribarra, D. T. (2021) · SpringerBriefs in psychology · doi:10.1007/978-3-030-74025-2 · Cited by 38 · see citing works
Evaluation of Some Coefficients for Use in Numerical Taxonomy of Microorganisms
Austin, B. P., & Colwell, R. R. (1977) · International Journal of Systematic Bacteriology · doi:10.1099/00207713-27-3-204 · Cited by 32 · see citing works
Notes on the H-measure of classifier performance
Hand, D. J., & Anagnostopoulos, C. (2022) · Advances in Data Analysis and Classification · doi:10.1007/s11634-021-00490-3 · Cited by 16 · see citing works
Self-citationMets, A. (2018) · Foundations of Science · doi:10.1007/s10699-018-9567-x · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Bart Baesens, Veronique Van Vlasselaer, and Wouter Verbeke. 2015. Fraud Analytics using Descriptive, Predictive, and Social Network Techniques: A Guide to Data Science for Fraud Detection. John Wiley and Sons, Hoboken, New Jersey.
- Thomas Benton. 2001. Theoretical and empirical models. Ph. D. Dissertation. Department of Mathematics, Imperial College, London.
- Kendrick Boyd, Kevin H. Eng, and C. David Page. 2013. Area under the precision-recall curve: Point estimates and con- fidence intervals. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 451-466.
- Jiawei Han and Micheline Kamber. 2000. Data Mining: Concepts and Techniques (1st ed.). Morgan Kaufmann, San Francisco.
- David J. Hand, Heikki Mannila, and Padhraic Smyth. 2001. Principles of Data Mining. MIT Press.
- Wojtek J. Krzanowski and David J. Hand. 2009. ROC Curves for Continuous Data. CRC Press, New York.
- David M. W. Powers. 2011. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. International Journal of Machine Learning Technology 2, 1 (2011), 37-63.
- David M. W. Powers. 2019. What the F-measure doesn't measure: Features, flaws, fallacies and fixes. https://arxiv.org/ abs/1503.06410v2 · arXiv
Cited by
Sorted by citations
Van Calster, B., Collins, G. S., Vickers, A. J., Wynants, L., Kerr, K. F., Barreñada, L., Varoquaux, G., Singh, K., Moons, K. G., Hernandez-Boussard, T., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100916 · Cited by 112 · see citing works
AI Companions Reduce Loneliness
De Freitas, J., Uguralp, A. K., Uguralp, Z. O., & Stefano, P. (2024) · arXiv · doi:10.48550/arxiv.2407.19096 · Cited by 12 · see citing works
A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys
Luo, Y., Myers, A. D., Drlica-Wagner, A., Dematties, D., Borchani, S., Valdes, F., Dey, A., Schlegel, D., Zhou, R., & DESI Legacy Imaging Surveys Team (2026) · The Open Journal of Astrophysics · doi:10.33232/001c.158430 · Cited by 1 · see citing works
- Authors
- Christen, P., Hand, D. J., & Kirielle, N.
- Affiliations
- Australian National University (AU)
- Imperial College London (GB)
- Venue
- ACM Computing Surveys
- Published
- 2023-10-06
- Volume
- 56
- Issue
- 3
- Pages
- 1-24
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4382397550
- Cited by
- 236
- References
- 48
- Citations by year
- 2026: 75 · 2025: 111 · 2024: 46 · 2023: 4
- FWCI
- 22.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Keywords
- F-measure, classification performance measures, precision and recall, harmonic mean, information retrieval evaluation, supervised classification, multi-class classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
