CoChat
Sign Up
Cited by 1,412Medical Image Segmentation TechniquesBrain Tumor Detection and ClassificationRadiomics and Machine Learning in Medical Imaging

The DOI matches its Crossref record; 137 of the 146 references it cites were found in a registry or an index.

The Liver Tumor Segmentation Benchmark (LiTS)

Open access
Bilic, Patrick, Christ, Patrick, Li, Hongwei Bran, Vorontsov, Eugene, Ben-Cohen, Avi, Kaissis, Georgios, Szeskin, Adi, Jacobs, Colin, Mamani, Gabriel Efrain Humpire, Chartrand, Gabriel, Lohöfer, Fabian, Holch, Julian Walter, Sommer, Wieland, Hofmann, Felix, Hostettler, Alexandre, Lev-Cohain, Naama, Drozdzal, Michal, Amitai, Michal Marianne, Vivanti, Refael, Sosna, Jacob, Ezhov, Ivan, Sekuboyina, Anjany, Navarro, Fernando, Kofler, Florian, Paetzold, Johannes C., Shit, Suprosanna, Hu, Xiaobin, Lipková, Jana, Rempfler, Markus, Piraud, Marie, Kirschke, Jan, Wiestler, Benedikt, Zhang, Zhiheng, Hülsemeyer, Christian, Beetz, Marcel, Ettlinger, Florian, Antonelli, Michela, Bae, Woong, Bellver, Míriam, Bi, Lei, Chen, Hao, Chlebus, Grzegorz, Dam, Erik B., Dou, Qi, Fu, Chi-Wing, Georgescu, Bogdan, Giró-i-Nieto, Xavier, Gruen, Felix, Han, Xu, Heng, Pheng-Ann, Hesser, Jürgen, Moltz, Jan Hendrik, Igel, Christian, Isensee, Fabian, Jäger, Paul, Jia, Fucang, Kaluva, Krishna Chaitanya, Khened, Mahendra, Kim, Ildoo, Kim, Jae-Hun, Kim, Sungwoong, Kohl, Simon, Konopczynski, Tomasz, Kori, Avinash, Krishnamurthi, Ganapathy, Li, Fan, Li, Hongchao, Li, Junbo, Li, Xiaomeng, Lowengrub, John, Ma, Jun, Maier-Hein, Klaus, Maninis, Kevis-Kokitsi, Meine, Hans, Merhof, Dorit, Pai, Akshay, Perslev, Mathias, Petersen, Jens, Pont-Tuset, Jordi, Qi, Jin, Qi, Xiaojuan, Rippel, Oliver, Roth, Karsten, Sarasua, Ignacio, Schenk, Andrea, Shen, Zengming, Torres, Jordi, Wachinger, Christian, Wang, Chunliang, Weninger, Leon, Wu, Jianrong, Xu, Daguang, Yang, Xiaoping, Yu, Simon Chun-Ho, Yuan, Yading, Yue, Miao, Zhang, Liping, Cardoso, Jorge, Bakas, Spyridon, Braren, Rickmer, Heinemann, Volker, Pal, Christopher, Tang, An, Kadoury, Samuel, Soler, Luc, van Ginneken, Bram, Greenspan, Hayit, Joskowicz, Leo, Menze, Bjoern·2023·Medical Image Analysis
DOI 10.1016/j.media.2022.102680PMC PMC10631490
Open this paper in CoChat — free accountOpen PDF

Abstract

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.media.2022.102680

Cites

137 of 137 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. A.10. Performance w.r.t tumor size and number of tumors. The test dataset is clustered by the number of tumors (#T) and size of the largest tumor per volume. Overall, participating methods perform well on volumes with large tumors and worse for volumes with small tumors. Worst results are achieved in chase where single small tumors (<15 mm3) occur. Best results are achieved when volumes show less than 6 tumors with an overall tumor volume above 40 mm3.
Fig. A.10. Performance w.r.t tumor size and number of tumors. The test dataset is clustered by the number of tumors (#T) and size of the largest tumor per volume. Overall, participating methods perform well on volumes with large tumors and worse for volumes with small tumors. Worst results are achieved in chase where single small tumors (<15 mm3) occur. Best results are achieved when volumes show less than 6 tumors with an overall tumor volume above 40 mm3.CC BY-NC-ND · PubMed Central
Fig. B.11. Performance w.r.t HU value difference between tumor and non-tumor liver tissue. Two robust metrics are calculated to cluster the results on the test set. First, the HU value difference between liver and tumor is calculated using both regions’ robust median absolute deviation per volume. Further, the clusters are split up by the tumor HU value difference calculated by the difference of the 90th percentile and 10th percentile. Participating methods perform best for volumes showing higher contrast between liver and tumor. Especially in the case of the liver, HU values are 40–60 points higher than the liver. Worst results are achieved in cases where the contrast is below 20 HU value, including tumors having a lower HU value than the liver.
Fig. B.11. Performance w.r.t HU value difference between tumor and non-tumor liver tissue. Two robust metrics are calculated to cluster the results on the test set. First, the HU value difference between liver and tumor is calculated using both regions’ robust median absolute deviation per volume. Further, the clusters are split up by the tumor HU value difference calculated by the difference of the 90th percentile and 10th percentile. Participating methods perform best for volumes showing higher contrast between liver and tumor. Especially in the case of the liver, HU values are 40–60 points higher than the liver. Worst results are achieved in cases where the contrast is below 20 HU value, including tumors having a lower HU value than the liver.CC BY-NC-ND · PubMed Central
Fig. C.12. Split and merge errors where a prediction splits a reference lesion into more than one connected component or merges multiple reference components into one, respectively. Reference connected components are shown with a solid color and predicted as regions with a dashed boundary and hatched interior. One-to-one correspondence is shown in green. One-to-two (a), two-to-one (b), and two-to-three (c) correspondence in orange. False negative in gray.
Fig. C.12. Split and merge errors where a prediction splits a reference lesion into more than one connected component or merges multiple reference components into one, respectively. Reference connected components are shown with a solid color and predicted as regions with a dashed boundary and hatched interior. One-to-one correspondence is shown in green. One-to-two (a), two-to-one (b), and two-to-three (c) correspondence in orange. False negative in gray.CC BY-NC-ND · PubMed Central
Fig. C.13. Two examples (top and bottom) of the process to establish a correspondence between connected components in the reference and prediction masks. Reference: solid color; prediction: dashed boundary and hatched interior. Left: reference components merged if the same predicted component overlaps them. Right: predicted components are merged together if the same merged reference component overlaps them. Corresponding reference and predicted components share the same color (green, orange). An undetected reference component is shown in solid gray. During the merge of reference components (left), predicted components that do not have the most significant overlap with a reference component are left unmatched (gray, dashed, and hatched). Their mapping is completed during the merge of predicted components (right).
Fig. C.13. Two examples (top and bottom) of the process to establish a correspondence between connected components in the reference and prediction masks. Reference: solid color; prediction: dashed boundary and hatched interior. Left: reference components merged if the same predicted component overlaps them. Right: predicted components are merged together if the same merged reference component overlaps them. Corresponding reference and predicted components share the same color (green, orange). An undetected reference component is shown in solid gray. During the merge of reference components (left), predicted components that do not have the most significant overlap with a reference component are left unmatched (gray, dashed, and hatched). Their mapping is completed during the merge of predicted components (right).CC BY-NC-ND · PubMed Central
Fig. 1. Example from the LiTS dataset depicting a variety of shapes of on contrast-enhanced abdominal CT scans acquired. While most exams in the dataset contain only one lesion, a large group of patients with some (2–7) or many (10–12) lesions, as shown in the histogram calculated over the whole dataset.
Fig. 1. Example from the LiTS dataset depicting a variety of shapes of on contrast-enhanced abdominal CT scans acquired. While most exams in the dataset contain only one lesion, a large group of patients with some (2–7) or many (10–12) lesions, as shown in the histogram calculated over the whole dataset.CC BY-NC-ND · PubMed Central
Fig. 2. Scatter plots of methods’ performances considering: (a) both segmentation and detection, (b) both distance- and overlap-based metrics for three challenge events. We observe that not all the top-performing methods in three LiTS challenges achieved good scores on tumor detection. The behavior of distance- and overlap-based metrics is similar.
Fig. 2. Scatter plots of methods’ performances considering: (a) both segmentation and detection, (b) both distance- and overlap-based metrics for three challenge events. We observe that not all the top-performing methods in three LiTS challenges achieved good scores on tumor detection. The behavior of distance- and overlap-based metrics is similar.CC BY-NC-ND · PubMed Central
Fig. 3. Inter-rater agreement between the existing annotation and new annotation sets. R1 represented the rater for the existing consensus annotation of the LiTS dataset. R2 re-annotated 15 CT scans from scratch. R3 and R4 are board-certified radiologists who checked and corrected the annotations. Specifically, one board-certified radiologist (R3) reviewed and corrected existing annotations. R4 re-evaluated R3’s final annotations and corrected them. The inter-rater agreement was calculated by the Dice score per case between the pairs of two raters.
Fig. 3. Inter-rater agreement between the existing annotation and new annotation sets. R1 represented the rater for the existing consensus annotation of the LiTS dataset. R2 re-annotated 15 CT scans from scratch. R3 and R4 are board-certified radiologists who checked and corrected the annotations. Specifically, one board-certified radiologist (R3) reviewed and corrected existing annotations. R4 re-evaluated R3’s final annotations and corrected them. The inter-rater agreement was calculated by the Dice score per case between the pairs of two raters.CC BY-NC-ND · PubMed Central
Fig. 4. Dice and ASD scores of three top-performing teams over the three events.
Fig. 4. Dice and ASD scores of three top-performing teams over the three events.CC BY-NC-ND · PubMed Central
Fig. 5. Distribution of mean Dice and ASD scores of all submissions in the CodaLab platform from the year 2017 to the year 2022.
Fig. 5. Distribution of mean Dice and ASD scores of all submissions in the CodaLab platform from the year 2017 to the year 2022.CC BY-NC-ND · PubMed Central
Fig. 6. Tumor segmentation results of the ISBI–LiTS 2017 challenge. The reference annotation is marked with green contour, while the prediction is with blue contour. One could observe that the boundary of liver lesion is rather ambiguous.
Fig. 6. Tumor segmentation results of the ISBI–LiTS 2017 challenge. The reference annotation is marked with green contour, while the prediction is with blue contour. One could observe that the boundary of liver lesion is rather ambiguous.CC BY-NC-ND · PubMed Central
Fig. 7. Tumor segmentation results of the MICCAI–LiTS 2017 challenge. The reference annotation is marked with green contour, while the prediction is with blue contour. One could observe that it is highly challenging to segment the liver lesion with poor contrast.
Fig. 7. Tumor segmentation results of the MICCAI–LiTS 2017 challenge. The reference annotation is marked with green contour, while the prediction is with blue contour. One could observe that it is highly challenging to segment the liver lesion with poor contrast.CC BY-NC-ND · PubMed Central
Fig. 8. Tumor segmentation results with selected cases of the tumor segmentation analysis regarding low (<20) and high (40–60) HU value difference. Compared are reference annotation (green), best-performing teams from ISBI 2017 (purple), MICCAI 2017 (orange), and MICCAI 2018 (blue). We can observe that a low HU value difference (<20) between tumor and liver tissue poses a challenge for tumor segmentation.
Fig. 8. Tumor segmentation results with selected cases of the tumor segmentation analysis regarding low (<20) and high (40–60) HU value difference. Compared are reference annotation (green), best-performing teams from ISBI 2017 (purple), MICCAI 2017 (orange), and MICCAI 2018 (blue). We can observe that a low HU value difference (<20) between tumor and liver tissue poses a challenge for tumor segmentation.CC BY-NC-ND · PubMed Central
Fig. 9. Samples of segmentation and detection results for small liver tumor. Compared are reference annotation (green), best-performing teams from ISBI 2017 (purple), MICCAI 2017 (orange), and MICCAI 2018 (blue).
Fig. 9. Samples of segmentation and detection results for small liver tumor. Compared are reference annotation (green), best-performing teams from ISBI 2017 (purple), MICCAI 2017 (orange), and MICCAI 2018 (blue).CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 146 references

Research works printed but not linked: 5

  1. Ben-Dan, 2008, Liver tumor segmentation in CT images using probabilistic methods, vol. 41, 43
  2. Deng, X., Du, G., 2008. 3D segmentation in the clinic: A grand challenge II-liver tumor segmentation. In: MICCAI Workshop.
  3. Erickson, 2016, Radiology data from the cancer genome atlas liver hepatocellular carcinoma [TCGA-LIHC] collection, Cancer Imaging Arch.
  4. Hann, 2000, Diagnostic imaging approaches and relationship to hepatobiliary cancer staging and therapy, vol. 19, 94
  5. Heussel, 2007, Follow-up CT measurement of liver malignoma according to RECIST and WHO vs. volumetry, RoFo: Fortschr. Geb. Rontgenstr. Nuklearmed., 179, 958

Cited by

Sorted by citations

See all 1,412

Related

Metadata

Authors
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., Lohöfer, F., Holch, J. W., Sommer, W., Hofmann, F., Hostettler, A., Lev-Cohain, N., Drozdzal, M., Amitai, M. M., Vivanti, R., Sosna, J., Ezhov, I., Sekuboyina, A., Navarro, F., Kofler, F., Paetzold, J. C., Shit, S., Hu, X., Lipková, J., Rempfler, M., Piraud, M., Kirschke, J., Wiestler, B., Zhang, Z., Hülsemeyer, C., Beetz, M., Ettlinger, F., Antonelli, M., Bae, W., Bellver, M., Bi, L., Chen, H., Chlebus, G., Dam, E. B., Dou, Q., Fu, C.-W., Georgescu, B., Giró-i-Nieto, X., Gruen, F., Han, X., Heng, P.-A., Hesser, J., Moltz, J. H., Igel, C., Isensee, F., Jäger, P., Jia, F., Kaluva, K. C., Khened, M., Kim, I., Kim, J.-H., Kim, S., Kohl, S., Konopczynski, T., Kori, A., Krishnamurthi, G., Li, F., Li, H., Li, J., Li, X., Lowengrub, J., Ma, J., Maier-Hein, K., Maninis, K.-K., Meine, H., Merhof, D., Pai, A., Perslev, M., Petersen, J., Pont-Tuset, J., Qi, J., Qi, X., Rippel, O., Roth, K., Sarasua, I., Schenk, A., Shen, Z., Torres, J., Wachinger, C., Wang, C., Weninger, L., Wu, J., Xu, D., Yang, X., Yu, S. C.-H., Yuan, Y., Yue, M., Zhang, L., Cardoso, J., Bakas, S., Braren, R., Heinemann, V., Pal, C., Tang, A., Kadoury, S., Soler, L., van Ginneken, B., Greenspan, H., Joskowicz, L., & Menze, B.
Affiliations
  • Technical University of Munich (DE)
  • Guangdong University of Foreign Studies (CN)
  • Polytechnique Montréal (CA)
  • Tel Aviv University (IL)
  • TUM Klinikum (DE)
  • Hebrew University of Jerusalem (IL)
  • Radboud University Medical Center (NL)
  • Université de Montréal (CA)
  • German Cancer Research Center (DE)
  • LMU Klinikum (DE)
  • Institut de Recherche contre les Cancers de l’Appareil Digestif (FR)
  • Rafael Advanced Defense Systems (Israel) (IL)
  • Hadassah Medical Center (IL)
  • University of Zurich (CH)
  • Helmholtz Munich (DE)
  • Chinese University of Hong Kong (HK)
  • Brigham and Women's Hospital (US)
  • ETH Zurich (CH)
  • Nanjing Drum Tower Hospital (CN)
  • King's College London (GB)
  • Hong Kong University of Science and Technology (HK)
  • Barcelona Supercomputing Center (ES)
  • The University of Sydney (AU)
  • University of Copenhagen (DK)
  • Siemens Healthcare (United States) (US)
  • Heidelberg University (DE)
  • University of North Carolina at Chapel Hill (US)
  • Fraunhofer Institute for Digital Medicine (DE)
  • Chinese Academy of Sciences (CN)
  • Indian Institute of Technology Madras (IN)
  • Samsung Medical Center (KR)
  • Kakao Brain (South Korea) (KR)
  • University of California, Irvine (US)
  • University of Bremen (DE)
  • RWTH Aachen University (DE)
  • University of Hong Kong (HK)
  • University of Tübingen (DE)
  • Medizinische Hochschule Hannover (DE)
  • University of Illinois Urbana-Champaign (US)
  • KTH Royal Institute of Technology (SE)
  • Nvidia (United States) (US)
  • Nanjing University (CN)
  • Icahn School of Medicine at Mount Sinai (US)
Corresponding author
  • Li, Hongwei Bran
  • Li, Hongchao
ORCID
Venue
Medical Image Analysis
Publisher
Elsevier BV
Published
2023-02
Volume
84
Pages
102680
ISSN
1361-8415 (ISSN-L)
DOI
10.1016/j.media.2022.102680
PMC
PMC10631490
OpenAlex
W2910094941
Cited by
1,412
References
146
Citations by year
2026: 259 · 2025: 337 · 2024: 311 · 2023: 195
FWCI
77.21
Type
journal article
Language
en
Open access
Licence
Locations
  • Medical Image Analysis (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (acceptedVersion, cc-by-nc-nd)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion, cc-by-nc-sa)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion, cc-by-nc-sa)
  • IRIS eCampus Telematic University (Università degli Studi eCampus) (submittedVersion)
  • mediaTUM (Technical University of Munich) (submittedVersion)
  • Site cant be reached (submittedVersion, cc-by-nc-nd)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • PolyPublie (École Polytechnique de Montréal) (submittedVersion)
  • PuSH - Publication Server of Helmholtz Zentrum München (submittedVersion, cc-by-nc-nd)
  • UPCommons institutional repository (Universitat Politècnica de Catalunya) (submittedVersion, cc-by-nc-nd)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by-nc-nd)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by-nc-nd)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • ETH Zürich Research Collection
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Deutsche Forschungsgemeinschaft (SFB824, GSC81, subproject B12)
  • Universität Zürich
  • Fondation de l'Association des radiologistes du Québec (34939)
  • National Institutes of Health (1s10od023495-01)
  • Fonds de Recherche du Québec - Santé (34939, Junior 2)
  • National Cancer Institute (U01CA242871)
  • International Graduate School of Science and Engineering (GSC 81)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Brain Tumor Detection and Classification
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
liver tumor segmentation, liver segmentation, liver cancer, computed tomography, segmentation benchmark, primary liver tumors, secondary liver tumors, lesion detection, medical image segmentation
MeSH terms
Algorithms, Humans, Image Processing, Computer-Assisted, Liver, Liver Neoplasms, Retrospective Studies, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review