The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Optimization for Medical Image Segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index
Open accessAbstract
In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovász-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 4×
“Pixel-wise (weighted) cross-entropy loss is still frequently used in these papers [1], [8].”
MethodInfluentialBakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., Freymann, J. B., Farahani, K., & Davatzikos, C. (2017) · Scientific Data · doi:10.1038/sdata.2017.117 · Cited by 3,049
Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations
Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S., & Jorge Cardoso, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_28 · Cited by 2,829
Fully Convolutional Networks for Semantic Segmentation
Long, J., Shelhamer, E., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4038 · Cited by 2,489
(1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0056181 · Cited by 1,577
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R. T., Berger, C., Ha, S. M., Rozycki, M., et al. (2018) · Apollo - University of Cambridge Repository · doi:10.17863/cam.38755 · Cited by 1,441
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R. T., Berger, C., Ha, S. M., Rozycki, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.02629 · Cited by 1,441
Morphometric analysis of white matter lesions in MR images: method and validation
Zijdenbos, A. P., Dawant, B. M., Margolin, R. A., & Palmer, A. C. (1994) · IEEE Transactions on Medical Imaging · doi:10.1109/42.363096 · Cited by 1,265
Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks
Salehi, S. S. M., Erdogmus, D., & Gholipour, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67389-9_44 · Cited by 1,086
Convexity, Classification, and Risk Bounds
Bartlett, P. L., Jordan, M. I., & McAuliffe, J. D. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000000907 · Cited by 1,037
Berman, M., Triki, A. R., & Blaschko, M. B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00464 · Cited by 1,027
Self-citationSubmodular functions and convexity
Lovász, L. (1983) · Mathematical Programming The State of the Art · doi:10.1007/978-3-642-68874-4_10 · Cited by 964
Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation
Rahman, M. A., & Wang, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-50835-1_22 · Cited by 910
Neurocomputing: Foundations of Research
Anderson, J. A., & Rosenfeld, E. (1988) · Cited by 851
Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., & Maier-Hein, K. H. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11726-9_21 · Cited by 341
Neurocomputing: Foundations of Research
Mingolla, E., & Bullock, D. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90025-7 · Cited by 334
Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks
Chen, L., Bentley, P., & Rueckert, D. (2017) · NeuroImage: Clinical · doi:10.1016/j.nicl.2017.06.016 · Cited by 304
Kuijf, H. J., Casamitjana, A., Collins, D. L., Dadar, M., Georgiou, A., Ghafoorian, M., Jin, D., Khademi, A., Knight, J., Li, H., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2905770 · Cited by 292
Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice
Bertels, J., Eelbode, T., Berman, M., Vandermeulen, D., Maes, F., Bisschops, R., & Blaschko, M. B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_11 · Cited by 283
Self-citationArtificial Intelligence for Medical Image Analysis: A Guide for Authors and Reviewers
England, J. R., & Cheng, P. M. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20490 · Cited by 180
Optimal Decisions from Probabilistic Models: The Intersection-over-Union Case
Nowozin, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.77 · Cited by 163
Tobel, J. D., Radesh, P., Vandermeulen, D., & Thevissen, P. W. (2017) · The Journal of Forensic Odonto-Stomatology · Cited by 114
Self-citation(2012) · Springer eBooks · doi:10.1007/springerreference_178893 · Cited by 90
No New-NetPreprint
Isensee, F., Kickingereder, P., Wick, W., et al. (2018) · arXiv (Cornell University) · Cited by 39
The Lovász Hinge: A Novel Convex Surrogate for Submodular Losses
Yu, J., & Blaschko, M. B. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2883039 · Cited by 31
Self-citationA Convex Surrogate Operator for General Non-Modular Loss FunctionsPreprint
Yu, J., & Blaschko, M. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.03373 · Cited by 7
Self-citation
Cited by
Medical image segmentation using deep learning: A survey
Wang, R., Lei, T., Cui, R., Zhang, B., Meng, H., & Nandi, A. K. (2022) · IET Image Processing · doi:10.1049/ipr2.12419 · Cited by 892
Loss odyssey in medical image segmentation
Ma, J., Chen, J., Ng, M., Huang, R., Li, Y., Li, C., Yang, X., & Martel, A. L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102035 · Cited by 662
Metadata
- Authors
- Eelbode, T., Bertels, J., Berman, M., Vandermeulen, D., Maes, F., Bisschops, R., & Blaschko, M. B.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-11
- Volume
- 39
- Issue
- 11
- Pages
- 3679-3690
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W3035546112
- Cited by
- 420
- References
- 35
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.