CoChat

Evaluation metrics and statistical tests for machine learning

Open access
Rainio, Oona, Teuho, Jarmo, Klén, Riku · 2024 · Scientific Reports

The DOI matches its Crossref record; 58 of the 60 references it cites were found in a registry or an index.

Abstract

Research on different machine learning (ML) has become incredibly popular during the past few decades. However, for some researchers not familiar with statistics, it might be difficult to understand how to evaluate the performance of ML models and compare them with each other. Here, we introduce the most common evaluation metrics used for the typical supervised ML tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval. We explain how to choose a suitable statistical test for comparing models, how to obtain enough values of the metric for testing, and how to perform the test and interpret its results. We also present a few practical examples about comparing convolutional neural networks used to classify X-rays with different lung infections and detect cancer tumors in positron emission tomography images.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-56706-x

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 ROC curves computed from the binary predictions of a test set containing 300 chest X-rays with COVID-19 and 300 X-rays from healthy patients by the modified U-Net (in blue) and InceptionV3 (in gray), accompanied by a straight line equal to the theoretic ROC curve of a random binary classifier. The x-axis here uses sensitivity instead of the false positive rate but, since its values range from 1 to 0, the end result is a typical plot, not its reflection. The AUC values are 0.845 for the modified U-Net and 0.821 for InceptionV3. The values of other evaluation metrics are in Table 4.
Figure 1 ROC curves computed from the binary predictions of a test set containing 300 chest X-rays with COVID-19 and 300 X-rays from healthy patients by the modified U-Net (in blue) and InceptionV3 (in gray), accompanied by a straight line equal to the theoretic ROC curve of a random binary classifier. The x-axis here uses sensitivity instead of the false positive rate but, since its values range from 1 to 0, the end result is a typical plot, not its reflection. The AUC values are 0.845 for the modified U-Net and 0.821 for InceptionV3. The values of other evaluation metrics are in Table 4.CC BY · PubMed Central
Figure 2 The binary tumor mask predicted by U-Net CNN with maximum dimensionality of 128 (in blue) and the ground-truth tumor mask drawn by a physician (in white) for one transaxial slice from a PET image of a head and neck cancer patient. The image is 128 × 128 pixels and the predicted segmentation mask contains 181 TP pixels, 16156 TN pixels, 17 FP pixels, and 30 FN pixels. This gives us Dice of 0.885, IoU of 0.794, and overall pixel accuracy of 0.997.
Figure 2 The binary tumor mask predicted by U-Net CNN with maximum dimensionality of 128 (in blue) and the ground-truth tumor mask drawn by a physician (in white) for one transaxial slice from a PET image of a head and neck cancer patient. The image is 128 × 128 pixels and the predicted segmentation mask contains 181 TP pixels, 16156 TN pixels, 17 FP pixels, and 30 FN pixels. This gives us Dice of 0.885, IoU of 0.794, and overall pixel accuracy of 0.997.CC BY · PubMed Central
Figure 3 The possible tasks for a model, their evaluation metrics, the values of the evaluation metric that must be computed for each model before statistical testing, the potential questions a statistical test could answer in the situation, and the suitable test.
Figure 3 The possible tasks for a model, their evaluation metrics, the values of the evaluation metric that must be computed for each model before statistical testing, the potential questions a statistical test could answer in the situation, and the suitable test.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 58 references

Research works printed but not linked: 9

  1. Demšar, J. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006).
  2. Planche, B. & Andres, E. Hands-On Computer Vision with TensorFlow 2: Leverage Deep Learning to Create Powerful Image Processing Apps with TensorFlow 2.0 and Keras (Packt Publishing, 2019).
  3. Zhu, W., Zeng, N. & Wang, N. Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. In NESUG proceedings: health care and life sciences, Baltimore, Maryland 67, vol. 19 (2010).
  4. Small Casler, K. & Gawlik, K. (eds) Laboratory Screening and Diagnostic Evaluation: An Evidence-Based Approach (Springer, 2022).
  5. Lantz, B. Machine Learning with R: Learn Techniques for Building and Improving Machine Learning Models, from Data Preparation to Model Tuning, Evaluation, and Working with Big Data (Packt Publishing, 2023).
  6. Sørensen, T. A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on Danish commons. K. Dan. Vidensk. Selsk. 5(4), 1–34 (1948).
  7. van Rossum, G. & Drake, F. L. Python 3 Reference Manual (CreateSpace, 2009).
  8. Bartlett, M. S. Properties of sufficiency and statistical tests. Proc. R. Stat. Soc. Ser. A 160, 268–282 (1937).
  9. Fox, J. & Weisberg, S. An R Companion to Applied Regression 3rd edn. (Sage, 2019).

Cited by

See all 1,338

Metadata

Authors
Rainio, O., Teuho, J., & Klén, R.
Affiliations
University of Turku (FI)
Corresponding author
Rainio, Oona
Venue
Scientific Reports
Published
2024-03-13
Volume
14
Issue
1
Pages
6086-6086
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4392741075
Cited by
1,338
References
60
Citations by year
2026: 594 · 2025: 593 · 2024: 151
FWCI
197.04
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UTUPub (University of Turku) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Suomen Kulttuurirahasto
  • Jenny ja Antti Wihurin Rahasto
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
hypothesis testing, model comparison, supervised learning, binary classification, multi-class classification, multi-label classification, regression analysis, image segmentation, object detection, information retrieval, convolutional neural networks
MeSH terms
Machine Learning, Supervised Machine Learning, Image Processing, Computer-Assisted, Neural Networks, Computer, Positron-Emission Tomography

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation metrics and statistical tests for machine learning. https://app.cochat.ai/papers/10.1038/s41598-024-56706-x
The paper
Rainio, O., Teuho, J., & Klén, R. (2024). Evaluation metrics and statistical tests for machine learning. Scientific Reports. https://doi.org/10.1038/s41598-024-56706-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.