Loss odyssey in medical image segmentation
Open accessAbstract
The loss function is an important component in deep learning-based segmentation methods. Over the past five years, many loss functions have been proposed for various segmentation tasks. However, a systematic study of the utility of these loss functions is missing. In this paper, we present a comprehensive review of segmentation loss functions in an organized manner. We also conduct the first large-scale analysis of 20 general loss functions on four typical 3D segmentation tasks involving six public datasets from 10+ medical centers. The results show that none of the losses can consistently achieve the best performance on the four segmentation tasks, but compound loss functions (e.g. Dice with TopK loss, focal loss, Hausdorff distance loss, and boundary loss) are the most robust losses. Our code and segmentation results are publicly available and can serve as a loss function benchmark. We hope this work will also provide insights on new loss function development for the community.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931 · see citing works
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556 · see citing works
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_49 · Cited by 7,945 · see citing works
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781 · see citing works
Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations
Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S., & Jorge Cardoso, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_28 · Cited by 2,829 · see citing works
H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes
Li, X., Chen, H., Qi, X., Dou, Q., Fu, C., & Heng, P. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2845918 · Cited by 2,530 · see citing works
Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.-A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M. A., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2837502 · Cited by 2,431 · see citing works
Self-citationMultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmentation
Ibtehaz, N., & Rahman, M. S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.08.025 · Cited by 2,334 · see citing works
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014 · see citing works
International Conference on Learning Representations (ICLR)
Abbas, M., Kivinen, J., & Raiko, T. (2016) · Cited by 1,632 · see citing works
Medical Image Computing and Computer-Assisted Intervention
Yang, G., Hawkes, D. J., Rueckert, D., Noble, J. A., & Taylor, C. (2009) · Research Explorer (The University of Manchester) · Cited by 1,591 · see citing works
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara, R. T., Berger, C., Ha, S. M., Rozycki, M., et al. (2018) · Apollo - University of Cambridge Repository · doi:10.17863/cam.38755 · Cited by 1,441 · see citing works
Self-citationThe Liver Tumor Segmentation Benchmark (LiTS)
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102680 · Cited by 1,412 · see citing works
Self-citationThe Liver Tumor Segmentation Benchmark (LiTS)
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.04056 · Cited by 1,412 · see citing works
Self-citationDeep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support
(2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9 · Cited by 1,149 · see citing works
Visualizing the Loss Landscape of Neural Nets
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000461393 · Cited by 1,148 · see citing works
Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks
Salehi, S. S. M., Deniz Erdogmus, B., & Gholipour, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67389-9_44 · Cited by 1,137 · see citing works
Berman, M., Triki, A. R., & Blaschko, M. B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00464 · Cited by 1,114 · see citing works
Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation
Rahman, M. A., & Wang, Y. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-50835-1_22 · Cited by 1,015 · see citing works
A Novel Focal Tversky Loss Function With Improved Attention U-Net for Lesion Segmentation
Abraham, N., & Khan, N. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759329 · Cited by 982 · see citing works
Weighted Res-UNet for High-Quality Retina Vessel Segmentation
Xiao, X., Lian, S., Luo, Z., et al. (2018) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme.2018.00080 · Cited by 970 · see citing works
2016 Fourth International Conference on 3D Vision (3DV)
(2016) · doi:10.1109/3dv40526.2016 · Cited by 856 · see citing works
Simpson, A. L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., van Ginneken, B., Kopp‐Schneider, A., Landman, B. A., Litjens, G. J. S., Menze, B. H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09063 · Cited by 703 · see citing works
Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks
Gibson, E., Giganti, F., Hu, Y., Bonmati, E., Bandula, S., Gurusamy, K. S., Davidson, B. R., Pereira, S. P., Clarkson, M. J., & Barratt, D. C. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2806309 · Cited by 701 · see citing works
Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks
Karimi, D., & Salcudean, S. E. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2930068 · Cited by 656 · see citing works
Boundary loss for highly unbalanced segmentation
Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, É., Dolz, J., & Ayed, I. B. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101851 · Cited by 645 · see citing works
Heller, N. E., Isensee, F., Maier-Hein, K. H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101821 · Cited by 586 · see citing works
Self-citationFully Dense UNet for 2-D Sparse Photoacoustic Tomography Artifact Removal
Guan, S., Khan, A. A., Sikdar, S., & Chitnis, P. V. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2912935 · Cited by 538 · see citing works
Zhu, W., Huang, Y., Zeng, L., Chen, X., Liu, Y., Qian, Z. Y., Du, N., Fan, W., & Xie, X. S. (2018) · Medical Physics · doi:10.1002/mp.13300 · Cited by 511 · see citing works
3D U-Net: Learning Dense Volumetric Segmentation from Sparse AnnotationPreprint
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.06650 · Cited by 488 · see citing works
2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)
Supatto, W. (2020) · doi:10.1109/isbi45749.2020 · Cited by 472 · see citing works
Combo loss: Handling input and output imbalance in multi-organ segmentation
Taghanaki, S. A., Zheng, Y., Zhou, S. K., Georgescu, B., Sharma, P., Xu, D., Comaniciu, D., & Hamarneh, G. (2019) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2019.04.005 · Cited by 463 · see citing works
Eelbode, T., Bertels, J., Berman, M., et al. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3002417 · Cited by 423 · see citing works
Evaluation of algorithms for Multi-Modality Whole Heart Segmentation: An open-access grand challenge
Zhuang, X., Li, L., Payer, C., Štern, D., Urschler, M., Heinrich, M. P., Oster, J., Wang, C., Smedby, Ö., Bian, C., et al. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101537 · Cited by 369 · see citing works
Self-citationDeep Learning and Data Labeling for Medical Applications
Cornebise, J., Cardoso, J. S., Lu, Z., Loog, M., Nascimento, J. C., Papa, J. P., Belagiannis, V., Tavares, J. M. R. S., Bradley, A. P., Peter, L., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46976-8 · Cited by 351 · see citing works
Nikolov, S., Blackwell, S., Zverovitch, A., Mendes, R. L., Livne, M., Fauw, J. D., Patel, Y., Meyer, C., Askham, H., Romera‐Paredes, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04430 · Cited by 239 · see citing works
Hashemi, S. R., Salehi, S. S. M., Deniz Erdogmus, B., Prabhu, S. P., Warfield, S. K., & Gholipour, A. (2018) · IEEE Access · doi:10.1109/access.2018.2886371 · Cited by 216 · see citing works
3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes
Wong, K. C. L., Moradi, M., Tang, H., & Syeda-Mahmood, T. F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00931-1_70 · Cited by 200 · see citing works
Topology-Preserving Deep Image SegmentationPreprint
Hu, X., Li, F., Samaras, D., & Chen, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05404 · Cited by 164 · see citing works
Bridging Category-level and Instance-level Semantic Image SegmentationPreprint
Wu, Z., Shen, C., & van den Hengel, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06885 · Cited by 148 · see citing works
nnU-Net: Breaking the Spell on Successful Medical Image Segmentation.Preprint
Isensee, F., Petersen, J., Köhl, S., Jäger, P. F., & Maier-Hein, K. H. (2019) · arXiv (Cornell University) · Cited by 144 · see citing works
The Importance of Skip Connections in Biomedical Image SegmentationPreprint
Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., & Pal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04117 · Cited by 132 · see citing works
Deep Convolutional Encoder Networks for Multiple Sclerosis Lesion Segmentation
Brosch, T., Yoo, Y., Tang, L. Y. W., Li, D. K. B., Traboulsee, A. L., & Tam, R. C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_1 · Cited by 126 · see citing works
How Distance Transform Maps Boost Segmentation CNNs: An Empirical Study
Ma, J., Zhan, W., Zhang, Y., et al. (2020) · Medical Imaging with Deep Learning · Cited by 50 · see citing works
Self-citationRemoving segmentation inconsistencies with semi-supervised non-adjacency constraint
Ganaye, P.-A., Sdika, M., Triggs, B., & Benoit-Cattin, H. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101551 · Cited by 37 · see citing works
Brugnara, G., Isensee, F., Neuberger, U., Bonekamp, D., Petersen, J., Diem, R., Wildemann, B., Heiland, S., Wick, W., Bendszus, M., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06593-y · Cited by 35 · see citing works
Distance Map Loss Penalty Term for Semantic SegmentationPreprint
Calivá, F., Iriondo, C., Martinez, A. M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03679 · Cited by 24 · see citing works
Seo, H., Bassenne, M., & Xing, L. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3031913 · Cited by 24 · see citing works
Universal Loss Reweighting to Balance Lesion Size Inequality in 3D Medical Image Segmentation
Shirokikh, B., Shevtsov, A. E., Kurmukov, A. G., Dalechina, A., Krivov, E., Kostjuchenko, V. V., Голанов, А. В., & Belyaev, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59719-1_51 · Cited by 21 · see citing works
Yang, S., Kweon, J., & Kim, Y. (2019) · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Segment anything in medical images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024) · Nature Communications · doi:10.1038/s41467-024-44824-z · Cited by 2,913 · see citing works
Self-citationZhang, J., Petitjean, C., & Ainouz, S. (2022) · Journal of Imaging · doi:10.3390/jimaging8020023 · Cited by 21 · see citing works
- Authors
- Ma, J., Chen, J., Ng, M., Huang, R., Li, Y., Li, C., Yang, X., & Martel, A. L.
- Affiliations
- Nanjing University of Science and Technology (CN)
- University of Toronto (CA)
- Nanjing University (CN)
- Corresponding author
- Ma, Jun
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2021-07
- Volume
- 71
- Pages
- 102035
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W3136424010
- Cited by
- 662
- References
- 56
- Citations by year
- 2026: 91 · 2025: 164 · 2024: 147 · 2023: 125 · 2022: 104 · 2021: 29 · 2020: 1 · 2013: 1
- FWCI
- 36.28
- Type
- journal article
- Language
- en
- Open access
- green
- zenodo.org
- Licence
- Locations
- Medical Image Analysis (publishedVersion)
- PubMed (publishedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- medical image segmentation, 3D medical image segmentation, Dice loss, focal loss, boundary loss, compound loss function, loss function comparison
- MeSH terms
- Humans, Image Processing, Computer-Assisted, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review