Towards a guideline for evaluation metrics in medical image segmentation
Open accessAbstract
In the last decade, research on artificial intelligence has seen rapid growth with deep learning models, especially in the field of medical image segmentation. Various studies demonstrated that these models have powerful prediction capabilities and achieved similar results as clinicians. However, recent studies revealed that the evaluation in image segmentation studies lacks reliable model performance assessment and showed statistical bias by incorrect metric implementation or usage. Thus, this work provides an overview and interpretation guide on the following metrics for medical image segmentation evaluation in binary as well as multi-class problems: Dice similarity coefficient, Jaccard, Sensitivity, Specificity, Rand index, ROC curves, Cohen’s Kappa, and Hausdorff distance. Furthermore, common issues like class imbalance and statistical as well as interpretation biases in evaluation are discussed. As a summary, we propose a guideline for standardized medical image segmentation evaluation to improve evaluation quality, reproducibility, and comparability in the research field.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477 · see citing works
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440 · see citing works
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Taha, A. A., & Hanbury, A. (2015) · BMC Medical Imaging · doi:10.1186/s12880-015-0068-x · Cited by 3,022 · see citing works
Codella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620 · see citing works
Dataset of breast ultrasound images
Al-Dhabyani, W., Gomaa, M., Khaled, H., & Fahmy, A. (2020) · Data in Brief · doi:10.1016/j.dib.2019.104863 · Cited by 2,405 · see citing works
Bernal, J., Sánchez, F. J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., & Vilariño, F. (2015) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2015.02.007 · Cited by 2,191 · see citing works
Studies in Health Technology and Informatics
Aphinyanaphongs, Y., & Aliferis, C. (2007) · Cited by 1,247 · see citing works
A Novel Focal Tversky Loss Function With Improved Attention U-Net for Lesion Segmentation
Abraham, N., & Khan, N. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759329 · Cited by 982 · see citing works
Deep semantic segmentation of natural and medical images: a review
Taghanaki, S. A., Abhishek, K., Cohen, J., Cohen‐Adad, J., & Hamarneh, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09854-1 · Cited by 888 · see citing works
Enhanced Performance of Brain Tumor Classification via Tumor Region Augmentation and Partition
Cheng, J., Huang, W., Cao, S., Yang, R., Yang, W., Yun, Z., Wang, Z., & Feng, Q. (2015) · PLOS ONE · doi:10.1371/journal.pone.0140381 · Cited by 879 · cited 1× · see citing works
“[30,31] and cell nuclei detection in pathology microscopy from Caicedo et al.”
A Review of Deep-Learning-Based Medical Image Segmentation Methods
Liu, X., Song, L., Liu, S., & Zhang, Y. (2021) · Sustainability · doi:10.3390/su13031224 · Cited by 871 · see citing works
Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl
Caicedo, J. C., Goodman, A., Karhohs, K. W., Cimini, B. A., Ackerman, J., Haghighi, M., Heng, C., Becker, T., Doan, M., McQuin, C., et al. (2019) · Nature Methods · doi:10.1038/s41592-019-0612-7 · Cited by 809 · see citing works
Addressing bias in artificial intelligence in health care
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019) · JAMA · doi:10.1001/jama.2019.18058 · Cited by 764 · see citing works
Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks
Karimi, D., & Salcudean, S. E. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2930068 · Cited by 656 · see citing works
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324 · see citing works
Retrieval of Brain Tumors by Adaptive Spatial Pooling and Fisher Vector Representation
Cheng, J., Yang, W., Huang, M., Huang, W. F., Jiang, J., Zhou, Y., Yang, R., Zhao, J., Feng, Y., Feng, Q., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157112 · Cited by 298 · see citing works
Going Deep in Medical Image Analysis: Concepts, Methods, Challenges, and Future Directions
Altaf, F., Islam, S. M. S., Akhtar, N., & Janjua, N. K. (2019) · IEEE Access · doi:10.1109/access.2019.2929365 · Cited by 293 · see citing works
Variability and reproducibility in deep learning for medical image segmentation
Renard, F., Guedria, S., Palma, N. D., & Vuillerme, N. (2020) · Scientific Reports · doi:10.1038/s41598-020-69920-0 · Cited by 231 · see citing works
Aydin, O. U., Taha, A. A., Hilbert, A., Khalil, A. A., Galinović, I., Fiebach, J. B., Frey, D., & Madai, V. I. (2021) · European Radiology Experimental · doi:10.1186/s41747-020-00200-2 · Cited by 156 · see citing works
Implementing the DICOM Standard for Digital Pathology
Herrmann, M. D., Clunie, D., Fedorov, A., Doyle, S., Pieper, S. D., Klepeis, V. E., Le, L. P., Mutter, G. L., Milstone, D. S., Schultz, T. J., et al. (2018) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_42_18 · Cited by 155 · see citing works
Müller, D., Soto‐Rey, I., & Krämer, F. (2022) · IEEE Access · doi:10.1109/access.2022.3182399 · Cited by 151 · see citing works
Self-citationMüller, D., & Kramer, F. (2019) · BMC Medical Imaging · doi:10.1186/s12880-020-00543-7 · Cited by 149 · see citing works
Self-citationStatistical validation metric for accuracy assessment in medical image segmentation
Popovic, A., de la Fuente, M., Engelhardt, M., & Radermacher, K. (2007) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-007-0125-1 · Cited by 115 · see citing works
Challenges of medical image processing
Scholl, I., Aach, T., Deserno, T. M., & Kuhlen, T. (2010) · Computer Science - Research and Development · doi:10.1007/s00450-010-0146-9 · Cited by 104 · see citing works
Artificial intelligence for clinical decision support in neurology
Pedersen, M., Verspoor, K. M., Jenkinson, M., Law, M., Abbott, D. F., & Jackson, G. D. (2020) · Brain Communications · doi:10.1093/braincomms/fcaa096 · Cited by 89 · see citing works
Automated Chest CT Image Segmentation of COVID-19 Lung Infection based on 3D U-Net
Müller, D., Rey, I. S., & Kramer, F. (2020) · arXiv.org · doi:10.1016/j.imu.2021.100681 · Cited by 75 · see citing works
Self-citationShaikh, F., Dehmeshki, J., Bisdas, S., Roettger, D., Kubassova, O., Aziz, M., & Awan, O. A. R. (2020) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2020.05.006 · Cited by 73 · see citing works
Comparison of metrics for the evaluation of medical segmentations using prostate MRI dataset
Nai, Y., Teo, B. W., Tan, N. L., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104497 · Cited by 72 · see citing works
Deep Learning Techniques for Biomedical and Health Informatics
(2020) · Elsevier eBooks · doi:10.1016/c2018-0-04781-7 · Cited by 70 · see citing works
Potentials of AI in medical image analysis in Gastroenterology and Hepatology
Chen, H., & Sung, J. J. �. (2020) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.15327 · Cited by 50 · see citing works
Designing image segmentation studies: Statistical power, sample size and reference standard quality
Gibson, E., Hu, Y., Huisman, H. J., & Barratt, D. C. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.004 · Cited by 36 · see citing works
Multi-Disease Detection in Retinal Imaging Based on Ensembling Heterogeneous Deep Learning Models
Müller, D., Soto‐Rey, I., & Krämer, F. (2021) · Studies in health technology and informatics · doi:10.3233/shti210537 · Cited by 35 · see citing works
Self-citationDeep Learning for Clinical Decision Support Systems: A Review from the Panorama of Smart Healthcare
Kumar, E. S., & Jayadev, P. S. (2019) · Studies in big data · doi:10.1007/978-3-030-33966-1_5 · Cited by 32 · see citing works
Lessons learned in transitioning to AI in the medical imaging of COVID-19
Naqa, I. M. E., Hu, Q., Chen, W., Li, H., Fuhrman, J. D., Gorre, N., & Giger, M. L. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.s1.010902 · Cited by 29 · see citing works
Jeon, E., Peltonen, L. M., Block, L. J., Charlene, R., Tayaben, J. L., Nibber, R., Pruinelli, L., Lozada, P. E., Janine, S., Maxim, T., et al. (2021) · Studies in health technology and informatics · doi:10.3233/shti281 · Cited by 28 · see citing works
MISeval: A Metric Library for Medical Image Segmentation Evaluation
Müller, D., Hartmann, D., Meyer, P., et al. (2022) · Studies in health technology and informatics · doi:10.3233/shti220391 · Cited by 22 · see citing works
Self-citationError Metrics for Quantitative Evaluation of Medical Image Segmentation
Niessen, W. J., Bouma, C. J., Vincken, K. L., et al. (2000) · Computational imaging and vision · doi:10.1007/978-94-015-9538-4_22 · Cited by 21 · see citing works
Rethinking Semantic Segmentation Evaluation for Explainability and Model SelectionPreprint
Zhang, Y., Mehta, S., & Caspi, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08418 · Cited by 15 · see citing works
A Review of Methods and Applications of the ROC Curve in Clinical Trials
Kumar, R. V., & Antony, G. M. (2010) · Drug Information Journal · doi:10.1177/009286151004400602 · Cited by 11 · see citing works
COVID-19 Image Segmentation Based on Deep Learning and Ensemble Learning
Meyer, P., Müller, D., Soto‐Rey, I., et al. (2021) · Studies in health technology and informatics · doi:10.3233/shti210223 · Cited by 9 · see citing works
Self-citationMüller, D., Soto‐Rey, I., & Krämer, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11440 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [40]Liu X Song L Liu S Zhang Y A review of deep-learning-based medical image segmentation methods Sustain 2021 13 3 1 29
- [46]Aydin OU Taha AA Hilbert A Khalil AA Galinovic I Fiebach JB On the usage of average hausdorff distance for segmentation performance assessment: hidden error when used for ranking Eur Radiol Exp 2021 5 1 4 10.1186/s41747-020-00200-2 33474675 PMC7817746 · DOI
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
“32 Finally, the model’s performance needs to be put into context, correctly selecting valid baseline alternatives for comparison, such as already recognized diagnostic tests, and formally evaluating with statistical approaches the added value that the model may bring.”
Lessons learned from RadiologyNET foundation models for transfer learning in medical radiology
Napravnik, M., Hržić, F., Urschler, M., Miletić, D., & Štajduhar, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-05009-w · Cited by 15 · see citing works
Marey, A., Serdysnki, K. C., Killeen, B. D., Unberath, M., & Umair, M. (2024) · BJR|Artificial Intelligence · doi:10.1093/bjrai/ubae008 · Cited by 7 · see citing works
ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality
Cosarinsky, M., Billot, R., Mansilla, L., Jimenez, G., Gaggión, N., Fu, G., Tirer, T., & Ferrante, E. (2025) · arXiv
“Commonly used metrics include the Dice-Sørensen coe ffi cient (DSC) and other overlap-based measures [1], as well as distance-based metrics, such as the Hausdor ff distance or the average symmetric surface distance (ASSD) [2], [3].”
- Authors
- Müller, D., Soto-Rey, I., & Kramer, F.
- Affiliations
- University of Augsburg (DE)
- Corresponding author
- Müller, Dominik
- ORCID
- Müller, Dominik
- Venue
- BMC Research Notes
- Published
- 2022-06-20
- Volume
- 15
- Issue
- 1
- Pages
- 210-210
- ISSN
- 1756-0500 (ISSN-L)
- PMC
- PMC9208116
- OpenAlex
- W4283160212
- Cited by
- 668
- References
- 45
- Citations by year
- 2026: 188 · 2025: 238 · 2024: 147 · 2023: 89 · 2022: 6
- FWCI
- 10.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Research Notes (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion, cc-by)
- Funders
- Bundesministerium für Bildung und Forschung (FKZ01ZZ1804E)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- medical image segmentation, segmentation evaluation metrics, Dice coefficient, Jaccard similarity, sensitivity and specificity, Rand index, ROC curve, Cohen's kappa, Hausdorff distance, class imbalance, statistical bias, reproducibility, multi-class segmentation, binary segmentation
- MeSH terms
- Algorithms, Artificial Intelligence, Image Processing, Computer-Assisted, ROC Curve, Reproducibility of Results, Guidelines as Topic, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
