Why Cohen’s Kappa should be avoided as performance measure in classification
Open accessAbstract
We show that Cohen's Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cohen's Kappa coefficient is a useful metric for imbalanced datasets, but the interpretation of its value is problematic.
“Cohen’s Kappa is an useful metric for imbalanced datasets; however, the interpretation of its value is problematic [84].”
From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [84]
The Introduction says Kappa can be inadequate when class distributions are imbalanced, and the Conclusion says it cannot serve as a dependable classification measure in such settings. The paper does discuss difficulties interpreting Kappa, but it contradicts the claim that Kappa is useful for imbalanced datasets. (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,493 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
Measures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 12,153 · see citing works
Building Predictive Models in R Using the caret Package
Kühn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 9,986 · see citing works
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515 · see citing works
(2005) · PubMed · Cited by 7,011 · see citing works
Allouche, O., Tsoar, A., & Kadmon, R. (2006) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2006.01214.x · Cited by 5,518 · see citing works
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 2× · see citing works
“Matthews Correlation Coefficient MCC was first introduced in the binary case by B.W. Matthews [4] to assess the performance of protein secondary structure prediction, as the ϕ -coefficient, which is the measure of association obtained by discretization of the Pearson’s correlation coefficient for…”
MethodThe kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093 · see citing works
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · see citing works
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179 · cited 2× · see citing works
“This same obstacle is extensively studied in [20]-[22].”
BackgroundReliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853 · see citing works
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838 · see citing works
An assessment of support vector machines for land cover classification
Huang, C., Davis, L. S., & Townshend, J. R. G. (2002) · International Journal of Remote Sensing · doi:10.1080/01431160110040323 · Cited by 1,801 · see citing works
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,590 · see citing works
Comparing global vegetation maps with the Kappa statistic
Monserud, R. A., & Leemans, R. (1992) · Ecological Modelling · doi:10.1016/0304-3800(92)90003-w · Cited by 1,106 · see citing works
Reliability in content analysis: some common misconceptions and recommendations.
Krippendorff, K. (2004) · Human Communication Research · doi:10.1093/hcr/30.3.411 · Cited by 1,027 · see citing works
Duro, D. C., Franklin, S. E., & Dubé, M. G. (2011) · Remote Sensing of Environment · doi:10.1016/j.rse.2011.11.020 · Cited by 998 · see citing works
An experimental comparison of performance measures for classification
Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.010 · Cited by 889 · see citing works
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & de la Torre, F. N. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.47 · Cited by 828 · see citing works
Misinterpretation and misuse of the kappa statistic
MACLURE, M., & WILLETT, W. C. (1987) · American Journal of Epidemiology · doi:10.1093/aje/126.2.161 · Cited by 808 · see citing works
Comparing two K-category assignments by a K-category correlation coefficient
Gorodkin, J. (2004) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2004.09.006 · Cited by 545 · see citing works
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406 · see citing works
McLaughlin, M. L. (2012) · Routledge eBooks · doi:10.4324/9780203856208 · Cited by 386 · see citing works
Cameron, M. L., Briggs, K. K., & Steadman, J. R. (2003) · The American Journal of Sports Medicine · doi:10.1177/03635465030310012601 · Cited by 311 · see citing works
Measures of Association for Cross Classifications III: Approximate Sampling Theory
Goodman, L. A., & Kruskal, W. (1979) · Springer series in statistics · doi:10.1007/978-1-4612-9995-0_3 · Cited by 299 · see citing works
Diversity of decision-making models and the measurement of interrater agreement
Uebersax, J. S. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.1.140 · Cited by 254 · see citing works
Clinicians are right not to like Cohen's
de Vet, H. C. W., Mokkink, L. B., Terwee, C. B., Hoekstra, O. S., & Knol, D. L. (2013) · BMJ · doi:10.1136/bmj.f2125 · Cited by 251 · see citing works
Measures of Association for Cross Classifications III: Approximate Sampling Theory
Goodman, L. A., & Kruskal, W. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500850 · Cited by 248 · see citing works
Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., Hall, M. A., & Pal, C. (2016) · Morgan Kaufmann Publishers Inc. eBooks · Cited by 199 · see citing works
On Similarity Indices and Correction for Chance Agreement
Albatineh, A. N., Niewiadomska–Bugaj, M., & Mihalko, D. (2006) · Journal of Classification · doi:10.1007/s00357-006-0017-z · Cited by 162 · see citing works
Classification-algorithm evaluation: Five performance measures based onconfusion matrices
Forbes, A. D. (1995) · The Journal of Clinical Monitoring · doi:10.1007/bf01617722 · Cited by 147 · see citing works
Assumptions behind Intercoder Reliability Indices
Zhao, X., Liu, J. S., & Deng, K. (2013) · Annals of the International Communication Association · doi:10.1080/23808985.2013.11679142 · Cited by 113 · see citing works
Delta: A new measure of agreement between two raters
Andrés, A. M., & Femia, P. (2004) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711004849268 · Cited by 110 · see citing works
INTEROBSERVER AGREEMENT IN THE CLASSIFICATION OF CEREBRAL PALSY
Blair, E. M., & Stanley, F. J. (1985) · Developmental Medicine & Child Neurology · doi:10.1111/j.1469-8749.1985.tb14133.x · Cited by 100 · see citing works
Analysing Intraclass Correlation for Dichotomous Variables
Mak, T. K. (1988) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347309 · Cited by 87 · see citing works
On Similarity Coefficients for 2×2 Tables and Correction for Chance
Warrens, M. J. (2008) · Psychometrika · doi:10.1007/s11336-008-9059-y · Cited by 83 · see citing works
MEASURING AGREEMENT WHEN TWO OBSERVERS CLASSIFY PEOPLE INTO CATEGORIES NOT DEFINED IN ADVANCE
Brennan, R. L., & Light, R. J. (1974) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1974.tb00535.x · Cited by 74 · see citing works
A Formal Proof of a Paradox Associated with Cohen’s Kappa
Warrens, M. J. (2010) · Journal of Classification · doi:10.1007/s00357-010-9060-x · Cited by 64 · see citing works
Powers, D. M. W. (2012) · Flinders Academic Commons (Flinders University) · doi:10.13140/rg.2.1.3492.0487 · Cited by 51 · see citing works
Association, agreement, and equity
Krippendorff, K. (1987) · Quality & Quantity · doi:10.1007/bf00167603 · Cited by 49 · see citing works
Interpretation of low kappa values
Donker, D. K., Hasman, A., & van Geijn, H. P. (1993) · International Journal of Bio-Medical Computing · doi:10.1016/0020-7101(93)90059-f · Cited by 48 · see citing works
Claessen, F. M. A. P., van den Ende, K. I. M., Doornberg, J. N., Guitton, T. G., Eygendaal, D. E., van den Bekerom, M. P. J., van der Lugt, J., Schep, N. W. L., Boerboom, A. L., van der Pluim, M., et al. (2015) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2015.03.029 · Cited by 39 · see citing works
Classification for Single-Trial N170 During Responding to Facial Picture With Emotion
Tian, Y., Zhang, H., Pang, Y., & Lin, J. (2018) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2018.00068 · Cited by 25 · see citing works
Bexkens, R., Claessen, F. M. A. P., Kodde, I. F., Oh, L. S., Eygendaal, D. E., & van den Bekerom, M. P. J. (2018) · Shoulder & Elbow · doi:10.1177/1758573218791813 · Cited by 20 · see citing works
A fast and efficient gene-network reconstruction method from multiple over-expression experiments
Stokić, D., Hanel, R., & Thurner, S. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-253 · Cited by 17 · see citing works
Immunological assays employed for the elucidation of an histoplasmosis outbreak in São Paulo, SP
Passos, A. N., Kohara, V. S., de Freitas, R. S., & Vicentini, A. P. (2014) · Brazilian Journal of Microbiology · doi:10.1590/s1517-83822014000400028 · Cited by 11 · see citing works
Reconstructing Linear Gene Regulatory Networks
Supper, J., Spieth, C., & Zell, A. (2007) · Springer eBooks · doi:10.1007/978-3-540-71783-6_26 · Cited by 9 · see citing works
On Association Coefficients, Correction for Chance, and Correction for Maximum Value
Warrens, M. J. (2013) · Journal of modern mathematics frontier · doi:10.14355/jmmf.2013.0204.01 · Cited by 5 · see citing works
Flagged first, then by citations
2 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [38]Kuhn M., et al: Caret package. Journal of statistical software 28(5), 1– 26 (2008)27774042
Cited by
Sorted by citations
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
“Despite its popularity, in the learning context there are a number of issues causing the kappa measure to produce unreliable results (for instance, its high sensitivity to the distribution of the marginal totals [73–75]), stimulating research for more reliable alternatives [76].”
BackgroundGoEmotions: A Dataset of Fine-Grained Emotions
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.372 · Cited by 873 · see citing works
Towards a guideline for evaluation metrics in medical image segmentation
Müller, D., Soto-Rey, I., & Kramer, F. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06096-y · Cited by 668 · see citing works
Classification Model Evaluation Metrics
Vujović, Ž. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120670 · Cited by 624 · see citing works
Bentivoglio, R., Isufi, E., Jonkman, S. N., & Taormina, R. (2022) · Hydrology and earth system sciences · doi:10.5194/hess-26-4345-2022 · Cited by 465 · see citing works
Chicco, D., Warrens, M., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 365 · see citing works
Sabanayagam, C., Xu, D., Ting, D. S. W., Nusinovici, S., Banu, R., Hamzah, H., Lim, C., Tham, Y.-C., Cheung, C. Y., Tai, E. S., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30063-7 · Cited by 279 · see citing works
Diogo, V. S., Ferreira, H. A., Prata, D. P., & Initiative, A. D. N. (2022) · Alzheimer s Research & Therapy · doi:10.1186/s13195-022-01047-y · Cited by 189 · see citing works
Opitz, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00675 · Cited by 169 · see citing works
A review of model evaluation metrics for machine learning in genetics and genomics
Miller, C., Portlock, T., Nyaga, D. M., & O'Sullivan, J. M. (2024) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2024.1457619 · Cited by 158 · see citing works
Ågren, A. M., Larson, J., Paul, S. S., Laudon, H., & Lidberg, W. (2021) · Geoderma · doi:10.1016/j.geoderma.2021.115280 · Cited by 127 · see citing works
Dorol--Beauroy-Eustache, O., & Mishara, B. L. (2021) · Preventive Medicine · doi:10.1016/j.ypmed.2021.106684 · Cited by 125 · see citing works
A Deep Learning Approach for Human Activities Recognition From Multimodal Sensing Devices
Ihianle, I. K., Nwajana, A. O., Ebenuwa, S. H., Otuka, R. I., Owa, K., & Orisatoki, M. O. (2020) · IEEE Access · doi:10.1109/access.2020.3027979 · Cited by 125 · see citing works
Urban tree species classification using UAV-based multi-sensor data fusion and machine learning
Hartling, S., Sagan, V., & Maimaitijiang, M. (2021) · GIScience & Remote Sensing · doi:10.1080/15481603.2021.1974275 · Cited by 110 · see citing works
Yu, H., Cooper, A., & Infante, D. (2020) · Ecological Modelling · doi:10.1016/j.ecolmodel.2020.109202 · Cited by 96 · see citing works
A comparative study on online machine learning techniques for network traffic streams analysis
Shahraki, A., Abbasi, M., Taherkordi, A., & Jurcut, A. D. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.108836 · Cited by 105 · see citing works
Ghosh, S., Rao, G. R., & Thomas, T. (2021) · Energy storage materials · doi:10.1016/j.ensm.2021.05.024 · Cited by 102 · see citing works
Weighted kappa measures for ordinal multi-class classification performance
Yılmaz, A. E., & Demirhan, H. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110020 · Cited by 96 · see citing works
Domingos, L. C. F., Santos, P. E., Skelton, P. S. M., Brinkworth, R. S. A., & Sammut, K. (2022) · Sensors · doi:10.3390/s22062181 · Cited by 93 · see citing works
“Cohen’s Kappa is an useful metric for imbalanced datasets; however, the interpretation of its value is problematic [84].”
BackgroundPasrija, P., Jha, P. K., Upadhyaya, P., Khan, M. S., & Chopra, M. (2022) · Current Topics in Medicinal Chemistry · doi:10.2174/1568026622666220701091339 · Cited by 86 · see citing works
Using deep learning to value free-form text data for predictive maintenance
Cadavid, J. P. U., Samir Lamouri, A., Grabot, B., & Fortin, A. (2021) · International Journal of Production Research · doi:10.1080/00207543.2021.1951868 · Cited by 75 · see citing works
Olczak, J., Emilson, F., Razavian, A. S., Antonsson, T., Stark, A., & Gordon, M. (2020) · Acta Orthopaedica · doi:10.1080/17453674.2020.1837420 · Cited by 70 · see citing works
Pawłuszek-Filipiak, K., & Borkowski, A. (2020) · Remote Sensing · doi:10.3390/rs12183054 · Cited by 68 · see citing works
Interrater reliability estimators tested against true interrater reliabilities
Zhao, X., Feng, G. C., Ao, S. H., & Liu, P. L. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01707-5 · Cited by 79 · see citing works
Zhang, X., Akber, M. Z., & Zheng, W. (2022) · Journal of Building Engineering · doi:10.1016/j.jobe.2022.104997 · Cited by 59 · see citing works
Phinzi, K., Abriha, D., & Szabó, S. (2021) · Remote Sensing · doi:10.3390/rs13152980 · Cited by 55 · see citing works
Combining Machine Learning and Molecular Dynamics to Predict P-Glycoprotein Substrates
Esposito, C., Wang, S., Lange, U., Oellien, F., & Riniker, S. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00525 · Cited by 52 · see citing works
Ma, J., Guan, Y., Xing, F., Eltzov, E., Wang, Y., Li, X., & Tai, B. (2023) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2023.131030 · Cited by 52 · see citing works
Fair evaluation of classifier predictive performance based on binary confusion matrix
Vanacore, A., Pellegrino, M. S., & Ciardiello, A. (2022) · Computational Statistics · doi:10.1007/s00180-022-01301-9 · Cited by 51 · see citing works
BenchMetrics: a systematic benchmarking method for binary classification performance metrics
Canbek, G., Temizel, T. T., & Sağıroğlu, Ş. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06103-6 · Cited by 51 · see citing works
Hanna, M. G., Olson, N. H., Zarella, M. D., Dash, R. C., Herrmann, M. D., Furtado, L. V., Stram, M. N., Raciti, P. M., Hassell, L. A., Mays, A., et al. (2023) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2023-0042-cp · Cited by 51 · see citing works
FastForest: Increasing random forest processing speed while maintaining accuracy
Yates, D., & Islam, M. Z. (2020) · Information Sciences · doi:10.1016/j.ins.2020.12.067 · Cited by 50 · see citing works
Evaluating Eye Movement Event Detection: A Review of the State of the Art
Startsev, M., & Zemblys, R. (2022) · Behavior Research Methods · doi:10.3758/s13428-021-01763-7 · Cited by 48 · see citing works
Distilled GPT for source code summarization
Su, C.-Y., & McMillan, C. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00421-4 · Cited by 49 · see citing works
Insights From the Virtual Team Science: Rapid Deployment During COVID-19
Kilcullen, M., Feitosa, J., & Eduardo Salas, A. (2021) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720821991678 · Cited by 46 · see citing works
Selecting patients with HER2-low breast cancer: Getting out of the tangle
Baez‐Navarro, X., Salgado, R., Denkert, C., Lennerz, J. K., Penault-Llorca, F. M., Viale, G., Bartlett, J. M. S., & van Deurzen, C. H. M. (2022) · European Journal of Cancer · doi:10.1016/j.ejca.2022.08.022 · Cited by 42 · see citing works
Benati, G., Calcagni, F., Martellozzo, F., Ghermandi, A., & Langemeyer, J. (2024) · Ecosystem Services · doi:10.1016/j.ecoser.2023.101594 · Cited by 40 · see citing works
Liang, Z., & Martell, M. A. C. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.665946 · Cited by 40 · see citing works
Martín, A., Miró, J., Amadó, C., Ruz, F., Ruíz, A., Ortega, O., & Clavé, P. (2022) · Dysphagia · doi:10.1007/s00455-022-10547-w · Cited by 37 · see citing works
Valuing free-form text data from maintenance logs through transfer learning with CamemBERT
Cadavid, J. P. U., Grabot, B., Lamouri, S., Pellerin, R., & Fortin, A. (2020) · Enterprise Information Systems · doi:10.1080/17517575.2020.1790043 · Cited by 38 · see citing works
Barnes, L. D., Lee, K., Kempa-Liehr, A. W., & Hallum, L. E. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272167 · Cited by 36 · see citing works
Forest Fire Risk Zone Mapping of Eravikulam National Park in India
Pradeep, G. S., Patel, N., Kuriakose, S. L., Ajin, R. S., Oniga, V.-E., Rajaneesh, A., Mammen, P. C., Prasad, M. K., Nikhil, S., & Danumah, J. H. (2021) · Croatian journal of forest engineering · doi:10.5552/crojfe.2022.1137 · Cited by 36 · see citing works
Dhar, J., & Jodder, A. K. (2020) · Ingénierie des systèmes d information · doi:10.18280/isi.250502 · Cited by 35 · see citing works
Ravindran, K. K. G., della Monica, C., Atzori, G., Lambert, D., Hassanin, H., Revell, V. L., & Dijk, D. (2023) · JMIR mhealth and uhealth · doi:10.2196/46338 · Cited by 35 · see citing works
Empirical analysis of performance assessment for imbalanced classification
Gaudreault, J.-G., & Branco, P. (2024) · Machine Learning · doi:10.1007/s10994-023-06497-5 · Cited by 33 · see citing works
Augmenting assessment with AI coding of online student discourse: A question of reliability
Misiejuk, K., Kaliisa, R., & Scianna, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100216 · Cited by 33 · see citing works
A Comprehensive Review on Deep Synergistic Drug Prediction Techniques for Cancer
Kumar, V., & Dogra, N. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09617-3 · Cited by 33 · see citing works
Wood–Leaf Classification of Tree Point Cloud Based on Intensity and Geometric Information
Sun, J., Wang, P., Gao, Z., Liu, Z., Li, Y., Gan, X., & Liu, Z. (2021) · Remote Sensing · doi:10.3390/rs13204050 · Cited by 33 · see citing works
Sun, J., Zhang, Y., Weishan, Q., & Chai, G. (2022) · Remote Sensing · doi:10.3390/rs14153653 · Cited by 33 · see citing works
Alqarni, F., Sagheer, A., Alabbad, A., & Hamdoun, H. (2025) · Scientific Reports · doi:10.1038/s41598-025-99515-6 · Cited by 38 · see citing works
Metrics Related to Confusion Matrix as Tools for Conformity Assessment Decisions
Božić, D., Runje, B., Lisjak, D., & Kolar, D. (2023) · Applied Sciences · doi:10.3390/app13148187 · Cited by 30 · see citing works
Casagrande, A., Fabris, F., & Girometti, R. (2020) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-020-02261-2 · Cited by 30 · see citing works
Neto, E. J. D. M., & Clementino, M. B. M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272287 · Cited by 30 · see citing works
Murariu, M.-G., Dorobanțu, F. R., & Tarniceriu, D. G. (2023) · Electronics · doi:10.3390/electronics12091958 · Cited by 30 · see citing works
Computationally Identifying Funneling and Focusing Questions in Classroom Discourse
Alic, S., Demszky, D., Mancenido, Z., Liu, J., Hill, H., & Jurafsky, D. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.27 · Cited by 54 · see citing works
AI-Driven sleep staging from actigraphy and heart rate
Song, T.-A., Chowdhury, S. R., Malekzadeh, M., Harrison, S. L., Blackwell-Hoge, T., Redline, S. S., Stone, K. L., Saxena, R., Purcell, S., & Dutta, J. (2023) · PLoS ONE · doi:10.1371/journal.pone.0285703 · Cited by 29 · see citing works
Flood Hazard Risk Mapping Using a Pseudo Supervised Random Forest
Esfandiari, M., Abdi, G., Jabari, S., McGrath, H., & Coleman, D. (2020) · Remote Sensing · doi:10.3390/rs12193206 · Cited by 29 · see citing works
Spatial database of planted forests in East Asia
Abbasi, A. O., Tang, X., Harris, N. L., Goldman, E. D., Gamarra, J. G. P., Herold, M., Kim, H. S., Luo, W., Silva, C. A., Tchebakova, N. M., et al. (2023) · Scientific Data · doi:10.1038/s41597-023-02383-w · Cited by 28 · see citing works
Interpretation of Artificial Intelligence Models in Healthcare
Ardakani, A. A., Airom, O., Khorshidi, H., Bureau, N. J., Salvi, M., Molinari, F., & Acharya, U. R. (2024) · Journal of Ultrasound in Medicine · doi:10.1002/jum.16524 · Cited by 29 · see citing works
Uhl, J. H., & Leyk, S. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2022.113117 · Cited by 27 · see citing works
Guidelines for the prediction of species interactions through binary classification
Poisot, T. (2023) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14071 · Cited by 27 · see citing works
Brecheisen, Z. S., & Richter, D. D. (2021) · CATENA · doi:10.1016/j.catena.2021.105264 · Cited by 26 · see citing works
A semi-hard voting combiner scheme to ensemble multi-class probabilistic classifiers
Delgado, R. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02447-7 · Cited by 26 · see citing works
Self-citationAyre, K., Bittar, A., Kam, J., Verma, S., Howard, L. M., & Dutta, R. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253809 · Cited by 24 · see citing works
Ågren, A. M., Hasselquist, E. M., Stendahl, J., Nilsson, M. B., & Paul, S. S. (2022) · SOIL · doi:10.5194/soil-8-733-2022 · Cited by 24 · see citing works
Best practices for machine learning in antibody discovery and development
Wossnig, L., Furtmann, N., Buchanan, A. G., Kumar, S., & Greiff, V. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104025 · Cited by 25 · see citing works
Ravindran, K. K. G., della Monica, C., Atzori, G., Nilforooshan, R., Hassanin, H., Revell, V. L., & Dijk, D. (2025) · SLEEP · doi:10.1093/sleep/zsaf122 · Cited by 23 · see citing works
Awe, O. O., Opateye, G. O., Johnson, C. A. G., Tayo, O. T., & Dias, R. (2023) · STEAM-H: Science, Technology, Engineering, Agriculture, Mathematics & Health · doi:10.1007/978-3-031-41352-0_18 · Cited by 24 · see citing works
Collado, A. A., Carrasco‐Velar, R., García‐Pedrajas, N., & García, G. C. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00908 · Cited by 22 · see citing works
Diagnosis of Acute Poisoning using explainable artificial intelligence
Chary, M. A., Boyer, E. W., & Burns, M. M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104469 · Cited by 22 · see citing works
Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions
Cabot, .-L. H., Abadi, D., Fischer, A. H., & Shutova, E. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.165 · Cited by 25 · see citing works
Measurement of key performance indicators of user experience based on software requirements
Atoum, I. (2023) · Science of Computer Programming · doi:10.1016/j.scico.2023.102929 · Cited by 22 · see citing works
A call to interpret disagreement components during classification assessment
Pontius, R. G., Francis, T., & Millones, M. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2469830 · Cited by 21 · see citing works
Validity of the personality disorder diagnosis in the Swedish National Patient Register
Kouppis, E., & Ekselius, L. (2020) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13166 · Cited by 21 · see citing works
Jovanović, L., Bačanin, N., Šimić, V., Mani, J. P., Zivkovic, M. Z., & Šarac, M. (2023) · Soft Computing · doi:10.1007/s00500-023-09496-9 · Cited by 21 · see citing works
Non-animal models for blood–brain barrier permeability evaluation of drug-like compounds
Dehnbostel, F. O., Dixit, V. A., Preißner, R., & Banerjee, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-59734-9 · Cited by 20 · see citing works
A deep learning approach for lower back-pain risk prediction during manual lifting
Snyder, K., Thomas, B., Lu, M., Jha, R., Barim, M. S., Hayden, M. A., & Werren, D. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0247162 · Cited by 20 · see citing works
Filho, C. R. M., do Valle, R. F., Silva, M. M. A. P. D. M., Mendes, R. G., Rolim, G. D. S., Pissarra, T. C. T., de Melo, M. C., Valera, C. A., Pacheco, F. A. L., & Fernandes, L. F. S. (2023) · Sustainability · doi:10.3390/su15086949 · Cited by 20 · see citing works
Artificial Intelligence–Guided Segmentation and Path Planning Software for Transthoracic Lung Biopsy
Too, C. W., Fong, K. Y., Hang, G., Sato, T., Nyam, C. Q., Ng, K. W., Ng, K. W., Ng, W. L., & Kawai, T. (2024) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2024.02.006 · Cited by 20 · see citing works
Kishore, B. S. P. C., Kumar, A., Saikia, P., Lele, N. V., Pandey, A. C., Srivastava, P., Bhattacharya, B. K., & Khan, M. L. (2020) · Journal of Asia-Pacific Biodiversity · doi:10.1016/j.japb.2020.07.001 · Cited by 19 · see citing works
Lin, P., Moni, M. A., Gau, S. S., & Eapen, V. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.637022 · Cited by 19 · see citing works
Impact of model choice in predicting urban forest storm damage when data is uncertain
Lambert, C., Landry, S., Andreu, M. G., Koeser, A. K., Starr, G., & Staudhammer, C. L. (2022) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2022.104467 · Cited by 19 · see citing works
Pseudo-online framework for BCI evaluation: a MOABB perspective using various MI and SSVEP datasets
Carrara, I., & Papadopoulo, T. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad171a · Cited by 19 · see citing works
Moradi, E., Elsisi, M., Mahmoud, K., Lehtonen, M., & Darwish, M. M. F. (2025) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2025.110731 · Cited by 18 · see citing works
TractLearn: A geodesic learning framework for quantitative analysis of brain bundles
Attyé, A., Renard, F., Baciu, M. V., Roger, É., Lamalle, L., Dehail, P., Cassoudesalle, H., & Calamante, F. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117927 · Cited by 18 · see citing works
Comparison of algorithms for error prediction in manufacturing with automl and a cost-based metric
Gerling, A., Ziekow, H., Heß, A., Schreier, U., Seiffer, C., & Abdeslam, D. O. (2022) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-021-01890-0 · Cited by 18 · see citing works
Shortcomings and areas for improvement in digital pathology image segmentation challenges
Foucart, A., Debeir, O., & Decaestecker, C. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102155 · Cited by 18 · see citing works
Simeth, J., Jiang, J., Nosov, A., Wibmer, A., Zelefsky, M., Tyagi, N., & Veeraraghavan, H. (2023) · Medical Physics (Lancaster) · doi:10.1002/mp.16320 · Cited by 18 · see citing works
Lee, J. Y., Lee, G.-G., & Hong, H. (2023) · Journal of Science Education and Technology · doi:10.1007/s10956-023-10042-3 · Cited by 18 · see citing works
How do we identify flash droughts? A case study in Central European Croplands
Alencar, P. H. L., & Paton, E. N. (2022) · Hydrology research · doi:10.2166/nh.2022.003 · Cited by 17 · see citing works
Sun, Z., Liu, H., Huyan, J., Li, W., Guo, M., Hao, X., & Pei, L. (2021) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2021.124894 · Cited by 17 · see citing works
A Cardiotocographic Classification using Feature Selection: A comparative Study
Prasetyo, S. E., Prastyo, P. H., & Arti, S. (2021) · JITCE (Journal of Information Technology and Computer Engineering) · doi:10.25077/jitce.5.01.25-32.2021 · Cited by 19 · see citing works
Green Space Quality Analysis Using Machine Learning Approaches
Rustamov, J., Rustamov, Z., & Zaki, N. (2023) · Sustainability · doi:10.3390/su15107782 · Cited by 17 · see citing works
Griffith, D. M., Byrd, K. B., Anderegg, L. D. L., Allan, E., Gatziolis, D., Roberts, D. A., Yacoub, R., & Nemani, R. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215533120 · Cited by 16 · see citing works
Assessing the role of school-based sex education in sexual health behaviours: a systematic review
Niland, R., Flinn, C., & Nearchou, F. (2024) · Cogent Psychology · doi:10.1080/23311908.2024.2309752 · Cited by 16 · see citing works
Smart prediction and optimization of air quality index with artificial intelligence
Mubeen, M., He, S., Rahman, M. S., Wang, L., Zhang, X., Ahmed, B., He, Z., & Han, Y. (2025) · Journal of Environmental Sciences · doi:10.1016/j.jes.2025.02.041 · Cited by 15 · see citing works
Data-Driven Model for Estimating the Probability of Riverine Levee Breach Due to Overtopping
Flynn, S., Zamanian, S., Vahedifard, F., Shafieezadeh, A., & Schaaf, D. M. (2021) · Journal of Geotechnical and Geoenvironmental Engineering · doi:10.1061/(asce)gt.1943-5606.0002743 · Cited by 15 · see citing works
Herfurth, K., Harpaz, Y., Roesch, J., Mueller, N., Walther, K., Kaltenhäuser, M., Pauli, E., Goldstein, A., Hamer, H. M., Buchfelder, M., et al. (2022) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.996989 · Cited by 15 · see citing works
de Sena, S., Häggman, M., Ranta, J., Roienko, O., Ilén, E., Acosta, N., Salama, J., Kirjavainen, T., Stevenson, N. J., Airaksinen, M., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e33295 · Cited by 15 · see citing works
Comparing ϕ and the F-measure as performance metrics for software-related classifications
Lavazza, L., & Morasca, S. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10199-2 · Cited by 15 · see citing works
- Authors
- Delgado, R., & Tibau, X.-A.
- Affiliations
- Universitat Autònoma de Barcelona (ES)
- Corresponding author
- Delgado, Rosario
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-09-26
- Volume
- 14
- Issue
- 9
- Pages
- e0222916
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6762152
- OpenAlex
- W2975256032
- Cited by
- 324
- References
- 53
- Citations by year
- 2026: 22 · 2025: 47 · 2024: 53 · 2023: 60 · 2022: 69 · 2021: 50 · 2020: 22 · 2019: 1
- FWCI
- 16.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- Cohen's kappa, Matthews correlation coefficient, multi-class classification, class imbalance, confusion matrix, entropy (information theory), classifier comparison
- MeSH terms
- Supervised Machine Learning, Data Analysis, Classification, Models, Theoretical, Reproducibility of Results, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
