CoChat
Sign Up
Cited by 139Radiomics and Machine Learning in Medical ImagingMedical Image Segmentation TechniquesImage Retrieval and Classification Techniques

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.

Radiomics feature reproducibility under inter-rater variability in segmentations of CT images

Open access
Haarburger, Christoph, Müller-Franzes, Gustav, Weninger, Leon, Kuhl, Christiane, Truhn, Daniel, Merhof, Dorit·2020·Scientific Reports
DOI 10.1038/s41598-020-69534-6PMC PMC7391354
Open this paper in CoChat — free accountOpen PDF

Abstract

Identifying image features that are robust with respect to segmentation variability is a tough challenge in radiomics. So far, this problem has mainly been tackled in test–retest analyses. In this work we analyse radiomics feature reproducibility in two phases: first with manual segmentations provided by four expert readers and second with probabilistic automated segmentations using a recently developed neural network (PHiseg). We test feature reproducibility on three publicly available datasets of lung, kidney and liver lesions. We find consistent results both over manual and automated segmentations in all three datasets and show that there are subsets of radiomic features which are robust against segmentation variability and other radiomic features which are prone to poor reproducibility under differing segmentations. By providing a detailed analysis of robustness of the most common radiomics features across several datasets, we envision that more reliable and reproducible radiomic models can be built in the future based on this work.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-020-69534-6

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The following pipeline is set up for each of the three datasets: (1) The PHiSeg network is trained using CT images and corresponding expert annotations. After training, given an unseen tumour image (2), the network samples (N=25) possible segmentations for that image (3). Based on these segmentations for a single tumour, (N=25) possible radiomics feature vectors are extracted (4). Finally, feature variability across the possible segmentations is calculated (5).
Figure 1 The following pipeline is set up for each of the three datasets: (1) The PHiSeg network is trained using CT images and corresponding expert annotations. After training, given an unseen tumour image (2), the network samples (N=25) possible segmentations for that image (3). Based on these segmentations for a single tumour, (N=25) possible radiomics feature vectors are extracted (4). Finally, feature variability across the possible segmentations is calculated (5).CC BY · PubMed Central
Figure 2 Pairwise Dice scores between expert raters and between PHiSeg segmentations (left), expert raters (middle) and between PHiSeg segmentations (right) for the LIDC dataset.
Figure 2 Pairwise Dice scores between expert raters and between PHiSeg segmentations (left), expert raters (middle) and between PHiSeg segmentations (right) for the LIDC dataset.CC BY · PubMed Central
Figure 3 ICC on LIDC dataset between individual features and PHiSeg raters (blue) and expert raters (red) grouped by feature category and sorted by ICC.
Figure 3 ICC on LIDC dataset between individual features and PHiSeg raters (blue) and expert raters (red) grouped by feature category and sorted by ICC.CC BY · PubMed Central
Figure 4 ICC between individual features and PHiSeg raters for KiTS (a) and LiTS (b) datasets grouped by feature category and sorted by ICC.
Figure 4 ICC between individual features and PHiSeg raters for KiTS (a) and LiTS (b) datasets grouped by feature category and sorted by ICC.CC BY · PubMed Central
Figure 5 Examples for PHiSeg segmentations on LIDC (a), KiTS (b) and LiTS (c) datasets. Aggregations of all 25 segmentations generated by the neural network are denoted in the rightmost column, respectively. Note that four expert segmentations are only available for the LIDC dataset, while the other datasets only contain one expert segmentation each.
Figure 5 Examples for PHiSeg segmentations on LIDC (a), KiTS (b) and LiTS (c) datasets. Aggregations of all 25 segmentations generated by the neural network are denoted in the rightmost column, respectively. Note that four expert segmentations are only available for the LIDC dataset, while the other datasets only contain one expert segmentation each.CC BY · PubMed Central
Figure 6 Mean ICC across all three datasets by feature categories.
Figure 6 Mean ICC across all three datasets by feature categories.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Cited by

Sorted by citations

Items 101–139 of 139

Related

See all

Metadata

Authors
Haarburger, C., Müller-Franzes, G., Weninger, L., Kuhl, C., Truhn, D., & Merhof, D.
Affiliations
  • RWTH Aachen University (DE)
  • Universitätsklinikum Aachen (DE)
  • Fraunhofer Institute for Digital Medicine (DE)
Corresponding author
Haarburger, Christoph
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2020-07-29
Volume
10
Issue
1
Pages
12688-12688
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-020-69534-6
PMC
PMC7391354
OpenAlex
W3046015102
Cited by
139
References
38
Citations by year
2026: 24 · 2025: 20 · 2024: 17 · 2023: 28 · 2022: 26 · 2021: 22 · 2020: 2
FWCI
9.49
Type
journal article
Language
en
Update notices
correction (2021-11-16)
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
Funders
  • RWTH Aachen University
  • Interreg
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
radiomics, feature reproducibility, interobserver variability, segmentation variability, CT images, manual segmentation, automatic segmentation, lung lesions, renal lesions, liver lesions, feature robustness
MeSH terms
Automation, Humans, Neoplasms, Radiographic Image Interpretation, Computer-Assisted, Tomography, X-Ray Computed, Observer Variation, Databases, Factual, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review