Cited by 1,412Medical Image Segmentation TechniquesBrain Tumor Detection and ClassificationRadiomics and Machine Learning in Medical Imaging

The Liver Tumor Segmentation Benchmark (LiTS)

Open access
Bilic, Patrick, Christ, Patrick, Li, Hongwei Bran, Vorontsov, Eugene, Ben-Cohen, Avi, Kaissis, Georgios, Szeskin, Adi, Jacobs, Colin +101 more·2023·Medical Image Analysis

Abstract

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.media.2022.102680

Cites

137 of 137 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Different lesion appearances and annotation targets have been reported across tumor benchmarks.

    “different lesion appearances and annotation targets have been reported across tumor benchmarks [3,8,23]”
  • Tumor segmentation is less stable because lesions are not tied to a fixed anatomical shape or location and may be small, heterogeneous, or weakly contrasted.

    “Tumor segmentation is less stable because lesions are not tied to a fixed anatomical shape or location and may be small, heterogeneous, or weakly contrasted [3,8,23].”
  • OrganMNIST3D is a 3D dataset based on the same source as Organ{A,C,S}MNIST, using 3D bounding boxes processed into 28 × 28 × 28 for multi-class classification of 11 body organs.

    “The source of the OrganMNIST3D is the same as that of the Organ{A,C,S}MNIST. Instead of 2D images, we directly use the 3D bounding boxes and process the images into 28 × 28 × 28 to perform multi-class classification of 11 body organs.”

    From MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification (2023) · cites this paper as [30]

    The Methods > Dataset passage describes LiTS as a collection of abdominal CT scans, but does not identify it as the source of OrganMNIST3D or Organ{A,C,S}MNIST. Searches for those dataset names and for 3D bounding boxes, 28 × 28 × 28 images, and classification of 11 organs found no supporting passage in our copy. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

9 of the 146 references couldn’t be matched to a known paper.

Research works printed but not linked: 10

  1. Ben-Dan I, Shenhav E, 2008. Liver tumor segmentation in CT images using probabilistic methods. In: MICCAI Workshop, vol. 41, p. 43.
  2. Cleary K., 2017. MIDAS - original datasets. http://insight-journal.org/midas/item/view/1346. (Accessed 01 December 2019).
  3. Deng X, Du G, 2008. 3D segmentation in the clinic: A grand challenge II-liver tumor segmentation. In: MICCAI Workshop.
  4. Hann LE, Winston CB, Brown KT, Akhurst T, 2000. Diagnostic imaging approaches and relationship to hepatobiliary cancer staging and therapy. In: Seminars in Surgical Oncology, vol. 19, (no. 2), Wiley Online Library, pp. 94– 115. 11126385
  5. Heussel CP, Meier S, Wittelsberger S, Götte H, Mildenberger P, Kauczor H-U, 2007. Follow-up CT measurement of liver malignoma according to RECIST and WHO vs. volumetry. RoFo: Fortschr. Geb. Rontgenstr. Nuklearmed 179 (9), 958– 964. 17594629
  6. Nordlinger B, Guiguet M, Vaillant J-C, Balladur P, Boudjema K, Bachellier P, Jaeck D, 1996. Surgical resection of colorectal carcinoma metastases to the liver: A prognostic scoring system to improve case selection, based on 1568 patients. Cancer: Interdiscipl. Int. J. Am. Cancer Soc 77 (7), 1254– 1262. 8608500
  7. Shirokikh B, Shevtsov A, Kurmukov A, Dalechina A, Krivov E, Kostjuchenko V, Golanov A, Belyaev M, 2020. Universal loss reweighting to balance lesion size inequality in 3D medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 523– 532.
  8. Stewart BW, Wild CP, 2014. World cancer report 2014. In: World Heal. Organ.. Technical Report, WHO Press, World Health Organization, p. 630.
  9. Tang Y, Tang Y, Zhu Y, Xiao J, Summers RM, 2020. Ê Net: An edge enhanced network for accurate liver and tumor segmentation on CT scans. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 512– 522.
  10. Wang X, Han S, Chen Y, Gao D, Vasconcelos N, 2019. Volumetric attention for 3D medical image segmentation and detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 175– 184.

Cited by

Sorted by citations

Authors
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., Lohöfer, F., Holch, J. W., Sommer, W., Hofmann, F., Hostettler, A., Lev-Cohain, N., Drozdzal, M., Amitai, M. M., Vivanti, R., Sosna, J., Ezhov, I., Sekuboyina, A., Navarro, F., Kofler, F., Paetzold, J. C., Shit, S., Hu, X., Lipková, J., Rempfler, M., Piraud, M., Kirschke, J., Wiestler, B., Zhang, Z., Hülsemeyer, C., Beetz, M., Ettlinger, F., Antonelli, M., Bae, W., Bellver, M., Bi, L., Chen, H., Chlebus, G., Dam, E. B., Dou, Q., Fu, C.-W., Georgescu, B., Giró-i-Nieto, X., Gruen, F., Han, X., Heng, P.-A., Hesser, J., Moltz, J. H., Igel, C., Isensee, F., Jäger, P., Jia, F., Kaluva, K. C., Khened, M., Kim, I., Kim, J.-H., Kim, S., Kohl, S., Konopczynski, T., Kori, A., Krishnamurthi, G., Li, F., Li, H., Li, J., Li, X., Lowengrub, J., Ma, J., Maier-Hein, K., Maninis, K.-K., Meine, H., Merhof, D., Pai, A., Perslev, M., Petersen, J., Pont-Tuset, J., Qi, J., Qi, X., Rippel, O., Roth, K., Sarasua, I., Schenk, A., Shen, Z., Torres, J., Wachinger, C., Wang, C., Weninger, L., Wu, J., Xu, D., Yang, X., Yu, S. C.-H., Yuan, Y., Yue, M., Zhang, L., Cardoso, J., Bakas, S., Braren, R., Heinemann, V., Pal, C., Tang, A., Kadoury, S., Soler, L., van Ginneken, B., Greenspan, H., Joskowicz, L., & Menze, B.
Affiliations
  • Technical University of Munich (DE)
  • Guangdong University of Foreign Studies (CN)
  • Polytechnique Montréal (CA)
  • Tel Aviv University (IL)
  • TUM Klinikum (DE)
  • Hebrew University of Jerusalem (IL)
  • Radboud University Medical Center (NL)
  • Université de Montréal (CA)
  • German Cancer Research Center (DE)
  • LMU Klinikum (DE)
  • Institut de Recherche contre les Cancers de l’Appareil Digestif (FR)
  • Rafael Advanced Defense Systems (Israel) (IL)
  • Hadassah Medical Center (IL)
  • University of Zurich (CH)
  • Helmholtz Munich (DE)
  • Chinese University of Hong Kong (HK)
  • Brigham and Women's Hospital (US)
  • ETH Zurich (CH)
  • Nanjing Drum Tower Hospital (CN)
  • King's College London (GB)
  • Hong Kong University of Science and Technology (HK)
  • Barcelona Supercomputing Center (ES)
  • The University of Sydney (AU)
  • University of Copenhagen (DK)
  • Siemens Healthcare (United States) (US)
  • Heidelberg University (DE)
  • University of North Carolina at Chapel Hill (US)
  • Fraunhofer Institute for Digital Medicine (DE)
  • Chinese Academy of Sciences (CN)
  • Indian Institute of Technology Madras (IN)
  • Samsung Medical Center (KR)
  • Kakao Brain (South Korea) (KR)
  • University of California, Irvine (US)
  • University of Bremen (DE)
  • RWTH Aachen University (DE)
  • University of Hong Kong (HK)
  • University of Tübingen (DE)
  • Medizinische Hochschule Hannover (DE)
  • University of Illinois Urbana-Champaign (US)
  • KTH Royal Institute of Technology (SE)
  • Nvidia (United States) (US)
  • Nanjing University (CN)
  • Icahn School of Medicine at Mount Sinai (US)
Corresponding author
  • Li, Hongwei Bran
  • Li, Hongchao
Venue
Medical Image Analysis
Publisher
Elsevier BV
Published
2023-02
Volume
84
Pages
102680
ISSN
1361-8415 (ISSN-L)
OpenAlex
W2910094941
Cited by
1,412
References
146
Citations by year
2026: 259 · 2025: 337 · 2024: 311 · 2023: 195 · 2022: 100 · 2021: 123 · 2020: 71 · 2019: 16
FWCI
77.21
Type
journal article
Language
en
Open access
Locations
  • Medical Image Analysis (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (acceptedVersion, cc-by-nc-nd)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion, cc-by-nc-sa)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion, cc-by-nc-sa)
  • IRIS eCampus Telematic University (Università degli Studi eCampus) (submittedVersion)
  • mediaTUM (Technical University of Munich) (submittedVersion)
  • Site cant be reached (submittedVersion, cc-by-nc-nd)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • PolyPublie (École Polytechnique de Montréal) (submittedVersion)
  • PuSH - Publication Server of Helmholtz Zentrum München (submittedVersion, cc-by-nc-nd)
  • UPCommons institutional repository (Universitat Politècnica de Catalunya) (submittedVersion, cc-by-nc-nd)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by-nc-nd)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • ETH Zürich Research Collection
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Deutsche Forschungsgemeinschaft (SFB824, GSC81, subproject B12)
  • Universität Zürich
  • Fondation de l'Association des radiologistes du Québec (34939)
  • National Institutes of Health (1s10od023495-01)
  • Fonds de Recherche du Québec - Santé (34939, Junior 2)
  • National Cancer Institute (U01CA242871)
  • International Graduate School of Science and Engineering (GSC 81)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
liver tumor segmentation, liver segmentation, liver cancer, computed tomography, segmentation benchmark, primary liver tumors, secondary liver tumors, lesion detection, medical image segmentation
MeSH terms
Algorithms, Humans, Image Processing, Computer-Assisted, Liver, Liver Neoplasms, Retrospective Studies, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review