CoChat
Sign Up
Cited by 241Cell Image Analysis TechniquesMedical Image Segmentation TechniquesArtificial Intelligence in Healthcare and Education

Understanding metric-related pitfalls in image analysis validation

Open access
Reinke, Annika, Tizabi, Minu D., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Kavur, A. Emre, Rädsch, Tim, Sudre, Carole H. +62 more·2024·Nature Methods
DOI 10.1038/s41592-023-02150-0PMC PMC11181963
Open this paper in CoChat — free accountOpen PDF

Abstract

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41592-023-02150-0

Cites

36 of 37 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current acceptance testing of AI auto contouring algorithms relies on area- and distance-based metrics that measure overall contour agreement but say nothing about where the model is uncertain.

    “Acceptance testing of AI auto contouring algorithms typically involves both area‐based metrics (e.g., Dice Similarity Coefficient, Intersection over Union (IoU)) and distance‐based metrics (e.g., Hausdorff Distance) evaluated against reference standard contours. 11 , 12 , 13 , 14 These metrics are well established for assessing contour agreement, but they say nothing about where the model is confi”

    From Uncertainty quantification of U‐Net based segmentation tool using conformal prediction (2026) · cites this paper as [13]

    The Results section ([P2.4], page unavailable) says reference-based metrics compare algorithm output with a reference annotation. The Extended Data identifies Dice and IoU as overlap-based metrics, while Results ([P2.3], page unavailable) identifies Hausdorff distance as distance-based. These passages do not establish what acceptance testing of AI autocontouring typically uses or explicitly address whether these metrics show where a model is uncertain. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. [1]Taghanaki Saeid Asgari Abhishek Kumar Cohen Joseph Paul Cohen-Adad Julien Hamarneh Ghassan Deep semantic segmentation of natural and medical images: a review Artificial Intelligence Review 2021 54 1 137 178
  2. [3]Bai Min Urtasun Raquel Deep watershed transform for instance segmentation Proceedings of the IEEE conference on computer vision and pattern recognition 2017 5221 5229
  3. [4]Bamira D Picard MH Imaging: Echocardiology—assessment of cardiac structure and function 2018 Elsevier
  4. [5]Bandos Andriy I Rockette Howard E Song Tao Gur David Area under the free-response roc curve (froc) and a related summary index Biometrics 2009 65 1 247 256 18479482 PMC2776072
  5. [6]Beneš Miroslav Zitová Barbara Performance evaluation of image segmentation algorithms on microscopic image data Journal of microscopy 2015 257 1 65 85 25233873
  6. [8]Bishop Christopher M Nasrabadi Nasser M Pattern recognition and machine learning 2006 4 Springer
  7. [10]Cao Chang Chicco Davide Hoffman Michael M The mcc-f1 curve: a performance evaluation technique for binary classification arXiv preprint 2020 arXiv:2006.11278 · arXiv
  8. [13]Cheng Bowen Girshick Ross Dollár Piotr Berg Alexander C Kirillov Alexander Boundary iou: Improving object-centric image segmentation evaluation Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021 15334 15342
  9. [14]Chicco Davide Jurman Giuseppe The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation BMC genomics 2020 21 1 1 13 PMC6941312 31898477
  10. [15]Chicco Davide Tötsch Niklas Jurman Giuseppe The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation BioData mining 2021 14 1 1 22 The manuscript addresses the challenge of evaluating binary classifications. It compares MCC to other metrics, explaining their mathematical relationships and providing use cases where MCC offers more informative results 33541410 PMC7863449
  11. [20]Cybenko George O’Leary Dianne P Rissanen Jorma The Mathematics of Information Coding, Extraction and Distribution 1998 107 Springer Science & Business Media
  12. [21]Davis Jesse Goadrich Mark The relationship between precision-recall and roc curves Proceedings of the 23rd international conference on Machine learning 2006 233 240
  13. [22]De Brabandere Bert Neven Davy Van Gool Luc Semantic instance segmentation with a discriminative loss function arXiv preprint 2017 arXiv: 1708.02551 · arXiv
  14. [23]De Fauw Jeffrey Ledsam Joseph R Romera-Paredes Bernardino Nikolov Stanislav Tomasev Nenad Blackwell Sam Askham Harry Glorot Xavier O’Donoghue Brendan Visentin Daniel Clinically applicable deep learning for diagnosis and referral in retinal disease Nature medicine 2018 24 9 1342 1350 30104768
  15. [24]Delgado Rosario Tibau Xavier-Andoni Why cohen’s kappa should be avoided as performance measure in classification PloS one 2019 14 9 e0222916 31557204 PMC6762152
  16. [25]DeLong Elizabeth R DeLong David M Clarke-Pearson Daniel L Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach Biometrics 1988 837 845 3203132
  17. [28]Everingham Mark Zisserman Andrew Williams Christopher KI Van Gool Luc Allan Moray Bishop Christopher M Chapelle Olivier Dalal Navneet Deselaers Thomas Dorkó Gyuri The 2005 pascal visual object classes challenge Machine Learning Challenges Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers 2006 117 176 Springer
  18. [30]Everingham Mark Eslami SM Ali Van Gool Luc Williams Christopher KI Winn John Zisserman Andrew The pascal visual object classes challenge: A retrospective International journal of computer vision 2015 111 1 98 136
  19. [31]Ferrer Luciana Analysis and comparison of classification metrics arXiv preprint 2022 arXiv:2209.05355 · arXiv
  20. [32]Gao Naiyu Shan Yanhu Wang Yupei Zhao Xin Yu Yinan Yang Ming Huang Kaiqi Ssap: Single-shot instance segmentation with affinity pyramid Proceedings of the IEEE/CVF International Conference on Computer Vision 2019 642 651
  21. [33]Gneiting Tilmann Raftery Adrian E Strictly proper scoring rules, prediction, and estimation Journal of the American statistical Association 2007 102 477 359 378
  22. [37]Gruber Sebastian Buettner Florian Trustworthy deep learning via proper calibration errors: A unifying approach for quantifying the reliability of predictive uncertainty arXiv preprint 2022 arXiv:2203.07835 · arXiv
  23. [38]Guo Chuan Pleiss Geoff Sun Yu Weinberger Kilian Q On Calibration of Modern Neural Networks ICML 2017 10
  24. [39]Gurcan Metin N Madabhushi Anant Rajpoot Nasir Pattern recognition in histopathological images: An icpr 2010 contest International Conference on Pattern Recognition 2010 226 234 Springer
  25. [40]Hanley James A McNeil Barbara J The meaning and use of the area under a receiver operating characteristic (roc) curve Radiology 1982 143 1 29 36 7063747
  26. [41]Hastie Trevor Tibshirani Robert Friedman Jerome H Friedman Jerome H The elements of statistical learning: data mining, inference, and prediction 2009 2 Springer
  27. [43]Hirsch Peter Mais Lisa Kainmueller Dagmar Patchperpix for instance segmentation arXiv preprint 2020 arXiv:2001.07626 · arXiv
  28. [45]Huttenlocher Daniel P Klanderman Gregory A Rucklidge William J Comparing images using the hausdorff distance IEEE Transactions on pattern analysis and machine intelligence 1993 15 9 850 863
  29. [48]Kirillov Alexander He Kaiming Girshick Ross Rother Carsten Dollár Piotr Panoptic segmentation Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019 9404 9413
  30. [51]Krause Jonathan Gulshan Varun Rahimy Ehsan Karth Peter Widner Kasumi Corrado Greg S Peng Lily Webster Dale R Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy Ophthalmology 2018 125 8 1264 1272 29548646
  31. [52]Kuhn Harold W The hungarian method for the assignment problem Naval research logistics quarterly 1955 2 1–2 83 97
  32. [53]Kulikov Victor Lempitsky Victor Instance segmentation of biological images using harmonic embeddings Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 3843 3851
  33. [54]Kull Meelis Nieto Miquel Perello Kängsepp Markus Filho Telmo Silva Song Hao Flach Peter Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration Advances in neural information processing systems 2019 32
  34. [55]Kumar Ananya Liang Percy S Ma Tengyu Verified uncertainty calibration Advances in Neural Information Processing Systems 2019 32
  35. [56]Lennerz Jochen K Green Ursula Williamson Drew FK Mahmood Faisal A unifying force for the realization of medical ai npj Digital Medicine 2022 5 1 1 3 36380011 PMC9666657
  36. [61]Maška Martin Ulman Vladimír Svoboda David Matula Pavel Matula Petr Ederra Cristina Urbiola Ainhoa España Tomás Venkatesan Subramanian Balak Deepak MW A benchmark for comparison of cell tracking algorithms Bioinformatics 2014 30 11 1609 1617 24526711 PMC4029039
  37. [62]Matthews Brian W Comparison of the predicted and observed secondary structure of t4 phage lysozyme Biochimica et Biophysica Acta (BBA)-Protein Structure 1975 405 2 442 451 1180967
  38. [64]Naeini Mahdi Pakdaman Cooper Gregory Hauskrecht Milos Obtaining well calibrated probabilities using bayesian binning Twenty-Ninth AAAI Conference on Artificial Intelligence 2015 PMC4410090 25927013
  39. [65]Nai Ying-Hwey Teo Bernice W Tan Nadya L Sophie O’Doherty Stephenson Mary C Thian Yee Liang Chiong Edmund Reilhac Anthonin Comparison of metrics for the evaluation of medical segmentations using prostate mri dataset Computers in Biology and Medicine 2021 134 104497 34022486
  40. [67]Nikolov Stanislav Blackwell Sam Zverovitch Alexei Mendes Ruheena Livne Michelle De Fauw Jeffrey Patel Yojan Meyer Clemens Askham Harry Romera-Paredes Bernadino Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study Journal of Medical Internet Research 2021 23 7 e26151 34255661 PMC8314151
  41. [69]Pham Hung Viet Qian Shangshu Wang Jiannan Lutellier Thibaud Rosenthal Jonathan Tan Lin Yu Yaoliang Nagappan Nachiappan Problems and opportunities in training deep learning software systems: An analysis of variance Proceedings of the 35th IEEE/ACM international conference on automated software engineering 2020 771 783
  42. [70]Popordanoska Teodora Sayer Raphael Blaschko Matthew B A consistent and differentiable lp canonical calibration error estimator Advances in Neural Information Processing Systems 2022
  43. [71]Powers David Martin Ward The problem with kappa Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics 2012 345 355
  44. [73]Reinke Annika Eisenmann Matthias Tizabi Minu D Sudre Carole H Rädsch Tim Antonelli Michela Arbel Tal Bakas Spyridon Cardoso M Jorge Cheplygina Veronika Common limitations of image processing metrics: A picture story arXiv preprint 2021 arXiv:2104.05642 · arXiv
  45. [75]Rosenfeld Azriel Pfaltz John L Sequential operations in digital picture processing Journal of the ACM (JACM) 1966 13 4 471 494
  46. [76]Saha Anindo Bosma Joeran Linmans Jasper Hosseinzadeh Matin Huisman Henkjan Anatomical and diagnostic bayesian segmentation in prostate mri − should different clinical objectives mandate different loss functions? arXiv preprint 2021 arXiv:2110.12889 · arXiv
  47. [78]Shit Suprosanna Paetzold Johannes C Sekuboyina Anjany Ezhov Ivan Unger Alexander Zhylka Andrey Pluim Josien PW Bauer Ulrich Menze Bjoern H cldice-a novel topology-preserving loss function for tubular structure segmentation Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021 16560 16569
  48. [85]Vaicenavicius Juozas Widmann David Andersson Carl Lindsten Fredrik Roll Jacob Schön Thomas Evaluating model calibration in classification The 22nd International Conference on Artificial Intelligence and Statistics 2019 3459 3467 PMLR
  49. [86]Van Ginneken Bram Armato Samuel G III de Hoop Bartjan van Amelsvoort-van de Vorst Saskia Duindam Thomas Niemeijer Meindert Murphy Keelin Schilham Arnold Retico Alessandra Fantacci Maria Evelina Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the anode09 study Medical image analysis 2010 14 6 707 722 20573538
  50. [87]Van Rijsbergen C Information retrieval: theory and practice Proceedings of the Joint IBM/University of Newcastle upon Tyne Seminar on Data Base Systems 1979 79
  51. [89]Vickers Andrew J Elkin Elena B Decision curve analysis: a novel method for evaluating prediction models Medical Decision Making 2006 26 6 565 574 17099194 PMC2577036
  52. [90]Vickers Andrew J Van Calster Ben Steyerberg Ewout W Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests bmj 2016 352 PMC4724785 26810254
  53. [91]Wack David S Dwyer Michael G Bergsland Niels Di Perri Carol Ranza Laura Hussein Sara Ramasamy Deepa Poloni Guy Zivadinov Robert Improved assessment of multiple sclerosis lesion segmentation agreement via detection and outline error estimates BMC medical imaging 2012 12 1 1 10 22812697 PMC3428663
  54. [93]Widmann David Lindsten Fredrik Zachariah Dave Calibration tests in multi-class classification: A unifying framework Advances in Neural Information Processing Systems 2019 32
  55. [95]Yeghiazaryan Varduhi Voiculescu Irina An overview of current evaluation methods used in medical image segmentation 2015 Department of Computer Science, University of Oxford
  56. [97]Zhu Qiuming On the performance of matthews correlation coefficient (mcc) for imbalanced dataset Pattern Recognition Letters 2020 136 71 80

Cited by

Sorted by citations

Related

Authors
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Buettner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B. A., Farahani, K., Ferrer, L., Galdran, A., van Ginneken, B., Glocker, B., Godau, P., Hashimoto, D. A., Hoffman, M. M., Huisman, M., Isensee, F., Jannin, P., Kahn, C. E., Kainmueller, D., Kainz, B., Karargyris, A., Kleesiek, J., Kofler, F., Kooi, T., Kopp-Schneider, A., Kozubek, M., Kreshuk, A., Kurc, T., Landman, B. A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A. L., Meijering, E., Menze, B., Moons, K. G. M., Müller, H., Nichyporuk, B., Nickel, F., Petersen, J., Rafelski, S. M., Rajpoot, N., Reyes, M., Riegler, M. A., Rieke, N., Saez-Rodriguez, J., Sánchez, C. I., Shetty, S., Summers, R. M., Taha, A. A., Tiulpin, A., Tsaftaris, S. A., Van Calster, B., Varoquaux, G., Yaniv, Z. R., Jäger, P. F., & Maier-Hein, L.
Affiliations
  • German Cancer Research Center (DE)
  • King's College London (GB)
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • Indiana University Indianapolis (US)
  • Holon Institute of Technology (IL)
  • Goethe University Frankfurt (DE)
  • IT University of Copenhagen (DK)
  • Leibniz Institute for Analytical Sciences - ISAS (DE)
  • Broad Institute (US)
  • National Cancer Institute (US)
  • Consejo Nacional de Investigaciones Científicas y Técnicas (AR)
  • Imperial College London (GB)
  • University of Pennsylvania (US)
  • University Health Network (CA)
  • Radboud University Medical Center (NL)
  • Inserm (FR)
  • University of Potsdam (DE)
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • Institut de Chirurgie Guidée par l'Image (FR)
  • Essen University Hospital (DE)
  • Lunit (South Korea) (KR)
  • Masaryk University (CZ)
  • European Molecular Biology Laboratory (DE)
  • Vanderbilt University (US)
  • University of Toronto (CA)
  • UNSW Sydney (AU)
  • University of Zurich (CH)
  • Utrecht University (NL)
  • University of Geneva (CH)
  • University Medical Center Hamburg-Eppendorf (DE)
  • Allen Institute for Cell Science (US)
  • University of Warwick (GB)
  • University of Bern (CH)
  • Simula Metropolitan Center for Digital Engineering (NO)
  • Google (United States) (US)
  • National Institutes of Health (US)
  • TU Wien (AT)
  • Oulu University Hospital (FI)
  • University of Edinburgh (GB)
  • National Institute of Allergy and Infectious Diseases (US)
Corresponding author
  • Reinke, Annika
  • Tizabi, Minu D.
  • Maier-Hein, Klaus
  • Jäger, Paul F.
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2024-02
Volume
21
Issue
2
Pages
182-194
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/s41592-023-02150-0
PMC
PMC11181963
OpenAlex
W4353015179
Cited by
241
References
36
Citations by year
2026: 108 · 2025: 82 · 2024: 48 · 2023: 2 · 2022: 1
FWCI
12.21
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (acceptedVersion, other-oa)
  • PubMed Central (acceptedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by-nc)
  • Radboud Repository (Radboud University) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • HAL (submittedVersion)
  • Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
  • TECNALIA Publications (Fundación TECNALIA Research & Innovation) (submittedVersion, public-domain)
  • ArODES (HES-SO (https://www.hes-so.ch/)) (submittedVersion)
  • IRIS eCampus Telematic University (Università degli Studi eCampus) (submittedVersion)
  • Lirias (KU Leuven) (submittedVersion)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (submittedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • PuSH - Publication Server of Helmholtz Zentrum München (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by-nc)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • National Science Foundation (GM135019)
  • Silicon Valley Community Foundation (2020-225720)
  • Foundation for Cardiovascular Research
  • Canadian Institute for Advanced Research
  • Wellcome Trust (WT 203148/Z/16/Z), 203148, WT213038/Z/18/Z, 213038)
  • Alzheimer's Society (AS‐JF‐17‐011)
  • Cancer Research UK (C49297/A27294)
  • Royal Academy of Engineering (RCSRF1819\8\25)
  • European Commission (101002198, HORIZON2020, 101088594)
  • Agence Nationale de la Recherche (ANR-10-IAHU-0002, ANR-10–IAHU-02, ANR-10)
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (31274, 27294, 225720, 212939, 205320)
  • Bundesministerium für Bildung und Forschung (161L0272)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek
  • Sydäntutkimussäätiö
  • Oulun Yliopisto (336449, Profi6 336449)
  • Novo Nordisk Fonden (NNF21OC0068816)
  • Innosuisse - Schweizerische Agentur für Innovationsförderung
  • National Institutes of Health (LM2018129, 5u24ca215109-05, 2020-225720, GM135019)
  • Natural Sciences and Engineering Research Council of Canada (rgpin-2022-05134)
  • Engineering and Physical Sciences Research Council ([WT203148/Z/16/Z], WT213038/Z/18/Z)
  • Centre For Medical Engineering, King’s College London (WT 203148/Z/16/Z)
  • National Cancer Institute (U24CA180924, UH3CA225021, UG3CA225021, U24CA215109,, 3U24CA215109-02, U01CA242871, 4UH3-CA225021-03, U24CA279629)
  • National Institute of General Medical Sciences (P41GM135019)
  • National Institute of Neurological Disorders and Stroke (R01NS042645)
  • NIH Clinical Center
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
validation metrics, medical image analysis, Delphi consensus, expert consensus, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review