Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 2,920

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

Open access
Taha, Abdel Aziz, Hanbury, Allan·2015·BMC Medical Imaging
DOI 10.1186/s12880-015-0068-x
Open this paper in CoChat — free account

Abstract

BACKGROUND: Medical Image segmentation is an important image processing step. Comparing images to evaluate the quality of segmentation is an essential part of measuring progress in this research area. Some of the challenges in evaluating medical segmentation are: metric selection, the use in the literature of multiple definitions for certain metrics, inefficiency of the metric calculation implementations leading to difficulties with large volumes, and lack of support for fuzzy segmentation by existing metrics. RESULT: First we present an overview of 20 evaluation metrics selected based on a comprehensive literature review. For fuzzy segmentation, which shows the level of membership of each voxel to multiple classes, fuzzy definitions of all metrics are provided. We present a discussion about metric properties to provide a guide for selecting evaluation metrics. Finally, we propose an efficient evaluation tool implementing the 20 selected metrics. The tool is optimized to perform efficiently in terms of speed and required memory, also if the image size is extremely large as in the case of whole body MRI or CT volume segmentation. An implementation of this tool is available as an open source project. CONCLUSION: We propose an efficient evaluation tool for 3D medical image segmentation using 20 evaluation metrics and provide guidelines for selecting a subset of these metrics that is suitable for the data and the segmentation task.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12880-015-0068-x

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 6 references

Research works printed but not linked: 67

  1. Zou KH, Warfield SK, Baharatha A, Tempany C, Kaus MR, Haker SJ, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Academic Radiology. 2004; 11:178–89. · DOI · Paper page
  2. Zou KH, Wells WM, Kikinis R, Warfield SK. Three validation metrics for automated probabilistic image segmentation of brain tumours. Stat Med. 2004; 23:1259–82. · DOI
  3. Kennedy DN, Makris N, Verne SC, Worth AJ. Neuroanatomical segmentation in MRI: Technological objectives. IJPRAI. 1997; 11(8):1161–87.
  4. Warfield SK, Westin CF, Guttmann CRG, Albert MS, Jolesz FA, Kikinis R. Fractional segmentation of white matter. In: Proceedings of Second International Conference on Medical Imaging Computing and Computer Assisted Interventions: 1999. p. 62–71. doi: 10.1007/10704282_7 . · DOI
  5. Shi R, Ngan KN, Li S. The objective evaluation of image object segmentation quality. ACIVS. 2013; 8192:470–9. doi: 10.1007/978-3-319-02895-8_42 . · DOI
  6. Fenster A, Chiu B. Evaluation of segmentation algorithms for medical imaging. In: Conf Proc IEEE Eng Med Biol Soc. Shanghai: 2005. p. 7186–189. · doi:10.1109/iembs.2005.1616166
  7. Vinh NX, Epps J, Bailey J. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. J Mach Learn Res. 2010; 11:2837–854.
  8. Gerig G, Jomier M, Chakos M. Valmet: A new validation tool for assessing and improving 3D object segmentation. In: Proceedings of the 4th International Conference on Medical Image Computing and Computer-Assisted Intervention: 2001. p. 516–23. doi: 10.1007/3-540-45468-3_62 . · DOI · Paper page
  9. Jr. Maurer CR, Qi R, Raghavan V. A linear time algorithm for computing exact Euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans Pattern Anal Mach Intell. 2003; 25(2):265–70. · DOI · Paper page
  10. Udupa JK, LeBlanc VR, Zhuge Y, Imielinska C, Schmidt H, Currie LM, et al. A framework for evaluating image segmentation algorithms. Comput Med Imaging Graph. 2006; 30(2):75–87. · DOI · Paper page
  11. Klement EP, Pap E, Mesiar R. Trends in logic. Netherlands: Springer; 2000. https://books.google.at/books?id=9HawzJbmXHUC .
  12. Campello RJGB. A fuzzy extension of the Rand index and other related indexes for clustering and classification assessment. Pattern Recognit Lett. 2007; 28(7):833–41. · DOI
  13. Dice LR. Measures of the amount of ecologic association between species. Ecology. 1945; 26(3):297–302. · DOI · Paper page
  14. Jaccard P. The distribution of the flora in the alpine zone. New Phytologist. 1912; 11(2):37–50. · DOI · Paper page
  15. Cardenes R, de Luis-Garcia R, Bach-Cuadra M. A multidimensional segmentation evaluation for medical image data. Comput Methods Prog Biomed. 2009; 96(2):108–24. · DOI
  16. Al-Faris AQ, Ngah UK, Isa NAM, Shuaib IL. MRI breast skin-line segmentation and removal using integration method of level set active contour and morphological thinning algorithms. J Med Sci. 2013. doi: 10.3923/jms.2012.286.291 . · doi:10.3923/jms.2012.286.291
  17. Chinchor N. MUC-4 evaluation metrics. In: Proceedings of the 4th Conference on Message Understanding: 1992. p. 22–9. doi: 10.3115/1072064.1072067 . · DOI · Paper page
  18. Rijsbergen CJV. Information Retrieval, 2nd edn. Newton, MA, USA: Butterworth-Heinemann; 1979.
  19. Martin DR, Fowlkes C, Tal D, Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. 8th Int’l Conf. Computer Vision. Vancouver, BC: 2001. p. 416–23. · DOI · Paper page
  20. Reddy AR, Prasad EV, Reddy LSS. Abnormality detection of brain mr image segmentation using iterative conditional mode algorithm. Int J Appl Inform Syst. 2013; 5(2):56–66.
  21. Vadaparthi N, Yarramalle S, Penumatsa SV, Murthy PSR. Segmentation of brain mr images based on finite skew gaussian mixture model with fuzzy c-means clustering and em algorithm. Int J Comput Appl. 2011; 28(10):18–26.
  22. Reddy AR, Prasad EV, Reddy LSS. Comparative analysis of brain tumor detection using different segmentation techniques. Int J Comput Appl. 2013; 82(14):14–28.
  23. Brennan RL, Light RJ. Measuring agreement when two observers classify people into categories not defined in advance. Br J Math Stat Psychol. 1974; 27(2). doi: 10.1111/j.2044-8317.1974.tb00535.x . · DOI
  24. Hallermeier E, Rifqi M, Henzgen S, Senge R. Comparing fuzzy partitions: A generalization of the rand index and related measures. IEEE T Fuzzy Syst. 2012. doi: 10.1109/TFUZZ.2011.2179303 . · doi:10.1109/tfuzz.2011.2179303
  25. Brouwer RK. Extending the Rand, adjusted Rand and Jaccard indices to fuzzy partitions. J Intell Inf Syst. 2009; 32(3):213–35. · DOI
  26. Anderson DT, Bezdek JC, Popescu M, Keller JM. Comparing fuzzy, probabilistic, and possibilistic partitions. Trans Fuz Sys. 2010; 18(5):906–18. · DOI
  27. Rand WM. Objective criteria for the evaluation of clustering methods. J Am Stat Assoc. 1971; 66(336):846–50. · DOI · Paper page
  28. Cover TM, Thomas JA. Elements of Information Theory. New York, NY, USA: Wiley-Interscience; 1991. · DOI · Paper page
  29. Viola P, III Wells WM. Alignment by maximization of mutual information. international journal of computer. Int J Comput Vis. 1997; 24(2):137–54. · DOI · Paper page
  30. Russakoff DB, Tomasi C, Rohlfing T, Jr Maurer CR. Image similarity using mutual information of regions. In: 8th European Conference on Computer Vision, ECCV: 2004. p. 596–607. doi: 10.1007/978-3-540-24672-5_47 . · DOI · Paper page
  31. Meila M. Comparing clusterings by the variation of information. In: Learning Theory and Kernel Machines. Berlin Heidelberg: Springer: 2003. p. 173–87.
  32. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20:37–46. · DOI · Paper page
  33. Hanley JA, Mcneil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143:29–36. · DOI · Paper page
  34. Taha AA, Hanbury A. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans Pattern Analysis and Machine Intelligence. 2014. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7053955 .
  35. Zhang D, Lu G. Review of shape representation and discription techniques. Pattern Recognit. 2004; 37(1):1–19. · DOI · Paper page
  36. Huttenlocher DP, Klanderman GA, Rucklidge WA. Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell. 1993; 15:850–63. · DOI · Paper page
  37. Zhao J, Long C, Xiong S, Liu C, Yua Z. A new k nearest neighbors search algorithm using cell grids for 3d scattered point cloud. Electron Electrical Eng. 2014; 20(1). doi: 10.5755/j01.eee.20.1.3926 . · doi:10.5755/j01.eee.20.1.3926
  38. Mahalanobis PC. On the generalised distance in statistics. In: Proceedings National Institute of Science, India: 1936. p. 49–55.
  39. McLachlan GJ. Mahalanobis distance. Resonance. 1999; 4:20–6. · DOI · Paper page
  40. Sladoje N, Lindblad J, Nystrom I. Defuzzification of spatial fuzzy sets by feature distance minimization. Image Vis Comput. 2011; 29:127–41. · doi:10.1016/j.imavis.2010.08.007
  41. Zwick R, Karlstein E, Budiscu DV. Measures of similarity among fuzzy concepts: a comparative analysis. Int J Approx Reason. 1987; 1(2):221–42. · DOI · Paper page
  42. Saporta G, Youness G. Comparing two partitions: Some proposals and experiments. In: Proceedings in Computational Statistics: 2002. p. 243–8. doi: 10.1007/978-3-642-57489-4_33 . · DOI
  43. Jain AK, Dubes RC. Algorithms for Clustering Data. Upper Saddle River, NJ, USA: Prentice-Hall, Inc.; 1988.
  44. Igual L, Soliva JC, Hernndez-Vela A, Escalera S, Vilarroya O, Radeva P. Supervised brain segmentation and classification in diagnostic of attention-deficit/hyperactivity disorder. In: HPCS. Madrid: 2012. p. 182–7. · DOI
  45. Fisher RA: Biological Monographs and Manuals. Statistical Methods for Research Workers. Edinburgh Scotland: Oliver and Boyd; 1970.
  46. Tustison NJ, Siqueira M, Gee JC. N-D linear time exact signed Euclidean distance transform. The Insight Journal. 2006. http://hdl.handle.net/1926/171 .
  47. Eric B, Andriy F, Nikos C. The use of robust local Hausdorff distances in accuracy assessment for image alignment of brain MRI. The Insight Journal. 2008. doi: 10.1007/978-3-540-89639-5_57 . · DOI
  48. Langs G, Mueller H, Menze BH, Hanbury A. Visceral: Towards large data in medical imaging - challenges and directions. In: MCBR-CDS MICCAI Workshop. Nice, France: 2013. p. 92–8. · DOI
  49. Jimenez del Toro OA, Goksel O, Menze B, Mueller H, Langs G, Weber MA, et al. Visceral - visual concept extraction challenge in radiology: Isbi 2014 challenge organization. In: Proceedings of the VISCERAL Challenge at ISBI. Beijing, China: 2014. p. 6–15.
  50. Fatourechi M, Ward RK, Mason SG, Huggins J, Schloegl A, Birch GE. Comparison of evaluation metrics in classification applications with imbalanced datasets. In: ICMLA. San Diego, CA: 2009. p. 777–82. · DOI
  51. Taha AA, Hanbury A, Jimenez del Toro O. A formal method for selecting evaluation metrics for image segmentation. In: 2014 IEEE International Conference on Image Processing (ICIP) (ICIP 2014). Paris, France: 2014. p. 932–6. · DOI
  52. Klein S, van der Heide UA, Raaymakers BW, Kotte ANTJ, Staring M, Pluim JPW. Segmentation of the prostate in mr images by atlas matching. In: ISBI. Arlington, VA: 2007. p. 1300–3. · doi:10.1109/isbi.2007.357098
  53. Cai X, Hou Y, Li C, Lee J, Wee WG. 2006. Evaluation of two segmentation methods on mri brain tissue structures. · doi:10.1109/iembs.2006.260725
  54. Gouttard S, Styner M, Prastawa M, Piven J, Gerig G. Assessment of reliability of multi-site neuroimaging via traveling phantom study. In: Proceedings of the 11th International Conference on Medical Image Computing and Computer-Assisted Intervention: 2008. p. 263–70. doi: 10.1007/978-3-540-85990-1_32 . · DOI
  55. Keyvan K, Mohammad Javad D, Kamran K, Mohammad Sadegh H, Kafshgari S. Comparison evaluation of three brain mri segmentation methods in software tools. In: Biomedical Engineering (ICBME). Isfahan: 2010. p. 1–4. · doi:10.1109/icbme.2010.5704955
  56. Babalola KO, Patenaude B, Aljabar P, Schnabel J, Kennedy D, Crum W, et al. Comparison and evaluation of segmentation techniques for subcortical structures in brain MRI. Med Image Comput Comput Assist Interv. 2008; 11(Pt 1):409–16.
  57. (Menze B, Jakab A, Bauer S, Reyes M, Prastawa M, Leemput KV, editors.)2012. MICCAI 2012 Challenge on Multimodal Brain Tumor Segmentation BRATS2012. Nice, France: MICCAI. http://www2.imm.dtu.dk/projects/BRATS2012/ .
  58. Khotanlou H, Colliot O, Atif J, Bloch I. 3D brain tumor segmentation in MRI using fuzzy classification, symmetry analysis and spatially constrained deformable models. Fuzzy Sets Syst. 2009; 160(10):1457–73. · DOI · Paper page
  59. Pang Y, Li L, Hu W, Peng Y, Liu L, Shao Y. Computerized segmentation and characterization of breast lesions in dynamic contrast-enhanced mr images using fuzzy c-means clustering and snake algorithm. Comput Math Methods Med. 2012. doi: 10.1155/2012/634907 . · DOI
  60. Yadav S, Meshram S. Brain tumor detection using clustering method. International Journal of Computational Engineering Research (IJCER). 2013:11–14. doi: 10.1016/j.eij.2015.01.003 . · DOI · Paper page
  61. Yadav S, Meshram S. Performance evaluation of basic segmented algorithms for brain tumor detection. J Electron Commun Eng IOSR. 2013; 5:08–13. · doi:10.9790/2834-560813
  62. Ginneken BV, Heimann T, Styner M. 3d segmentation in the clinic: A grand challenge. In: MICCAI Workshop on 3D Segmentation in the Clinic: 2007. p. 7–15. http://hdl.handle.net/10380/1509 .
  63. Wehrens R, Buydens LMC, Fraley C, Raftery AE. Model-based clustering for image segmentation and large datasets via sampling. J Classif. 2004; 21(2). doi: 10.1007/s00357-004-0018-8 . · DOI
  64. Moberts B, Vilanova A, van Wijk JJ. Evaluation of fiber clustering methods for diffusion tensor imaging. In: IEEE Visualization: 2005. p. 65–72. doi: 10.1109/VISUAL.2005.1532779 . · DOI
  65. Doring TM, Kubo TTA, Cruz LCH, Juruena MF, Fainberg J, Domingues RC, et al. Evaluation of hippocampal volume based on mr imaging in patients with bipolar affective disorder applying manual and automatic segmentation techniques. J Magn Reson Imaging. 2011; 33(3):565–72. · DOI
  66. Morain-Nicolier F, Lebonvallet S, Baudrier E, Ruan S. Hausdorff distance based 3D quantification of brain tumor evolution from MRI images. In: 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Lyon, France: 2007. p. 5597–600. · doi:10.1109/iembs.2007.4353615
  67. Narendran P, Narendira Kumar VK, Somasundaram K. 3D Brain Tumors and internal brain structures segmentation in mr images. Int J Image Graphics Signal Process. 2012; 1:ISSN: 2074-9074.

Cited by

See all 2,920

Metadata

Authors
Taha, A. A., & Hanbury, A.
Venue
BMC Medical Imaging
Publisher
Springer Science and Business Media LLC
Published
2015-08-12
Volume
15
Issue
1
Pages
29-29
ISSN
1471-2342 (ISSN-L)
DOI
10.1186/s12880-015-0068-x
OpenAlex
W1909740415
Cited by
2,920
References
6
FWCI
45.06
Type
journal article
Language
en
Open access
gold
Licence
www.springer.com/tdm (unspecified)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. https://app.cochat.ai/papers/10.1186/s12880-015-0068-x
The paper
Taha, A. A., & Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging. https://doi.org/10.1186/s12880-015-0068-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.