Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation

In the paper

“The authors show the effectiveness of the suggested framework over a pure supervised approach across multiple datasets.”

What the sources say

• Not checked yetcites Nie et al. (2018)

In the paper

“However, weights pre-trained via self-supervised learning tend to lose their edge over randomly initialized weights when the size of the training set changes from 25 to 400 images.”

What the sources say

• Not checked yetcites Ross et al. (2018)

In the paper

“The performance gain however drops to 1 point in Dice when the authors increase the size of labeled training set to 100 CT scans.”

What the sources say

• Not checked yetcites Zhou et al. (2018a)

In the paper

“The training set consists of 50 labeled CT and 100 unlabeled CT scans. Under this data split, the suggested method achieves on average four points increase in Dice over the model trained using only the labeled data.”

What the sources say

• Not checked yetcites Zhou et al. (2018a)

In the paper

“The suggested model is trained for the task of brain region segmentation using 12 labeled and 749 unlabeled MR images. On a tests set with 20 MR scans, the suggested model achieves 1.6 points increase in Dice over the base model trained using only labeled data.”

What the sources say

• Not checked yetcites Zhao et al. (2019b)

In the paper

“For pancreas segmentation in the NIH dataset (62 training and 20 test CT scans), the authors report seven points and five points increase in Dice over a full supervised model, when 10% and 20% of the training set is used as the initial labeled training set.”

What the sources say

• Not checked yetcites Xia et al. (2020)

In the paper

“In 3 iterations, the suggested method achieves 12-point increase in Dice over a model trained using the initial pseudo mask generated by K-means.”

What the sources say

• Not checked yetcites Zhang et al. (2018b)

In the paper

“The suggested method proves effective in segmenting brain tissues from MR images when the training set contains only a few annotated images.”

What the sources say

• Not checked yetcites Mondal et al. (2018)

In the paper

“For tumor segmentation in the BraTS dataset Menze et al. (2014) (120 training and 50 test MR scans), the authors report six and seven points increase in Dice over supervised model when 20 and 50 labeled MR images are used for training.”

What the sources say

• Not checked yetcites Chen et al. (2019c)
• Not checked yetcites Menze et al. (2014) source

In the paper

“The suggested semi-supervised framework proves promising in improving the segmentation of multiple sclerosis in the presence of limited data and domain shift.”

What the sources say

• Not checked yetcites Baur et al. (2017)

In the paper

“This improvement is however similar to that of other competing semi-supervised methods used for comparison (e.g., Li et al. (2019b)).”

What the sources say

• Not checked yetcites Yu et al. (2019)
• Not checked yetcites Li et al. (2019b)

In the paper

“The semi-supervised model, however, continues to outperform the supervised model by four points in Dice even when the entire large set of the labeled dataset is used for training-a significant finding rarely reported before.”

What the sources say

• Not checked yetcites Chen et al. (2019c)

In the paper

“On a test set with 200 images, the semi-supervised model achieves four points and one point increase in Dice over the supervised model, when 12.5% and 100% of the labeled set is used for training, respectively.”

What the sources say

• Not checked yetcites Sedai et al. (2017)

In the paper

“For the task of atrial segmentation in a dataset with 80 training and 20 test MR images, the authors report four points increase in Dice over a supervised model trained using 16 labeled images.”

What the sources say

• Not checked yetcites Yu et al. (2019)

In the paper

“The reactive methods tackle the problem of scarce annotation through a post segmentation refinement using variants of conditional random fields.”

In the paper

“The limited capability of the traditional methods in handling the problem of scarce annotations has led to the development of modern reactive and proactive approaches.”

In the paper

“The results show that the synthetic images generated by spatially constrained CycleGAN are more effective than CycleGAN in improving the performance of the base segmentation model.”

What the sources say

• Not checked yetcites Fu et al. (2018a)

In the paper

“Ronneberger et al. (2015) show that augmenting the training set with random elastic deformations is key to training a segmentation network with very few annotated images.”

What the sources say

• Not checked yetcites Ronneberger et al. (2015)

In the paper

“However, these techniques can only partially address the problem of limited annotation.”

In the paper

“Despite its simplicity, mixup has been highly effective for both natural and medical images.”

In the paper

“The results show that the universal network usually performed at par with or outperformed the specialized networks.”

What the sources say

• Not checked yetcites Harouni et al. (2018a)

In the paper

“The authors test the lung segmentation model using 34 images with peripheral nodules from the LIDC dataset, demonstrating that the model trained with the synthetic data achieves 2 points increase in Dice over the model trained using only real images.”

What the sources say

• Not checked yetcites Jin et al. (2018)

In the paper

“However, the performance gap is largely bridged in the presence of typical data augmentation.”

What the sources say

• Not checked yetcites Shin et al. (2018)

In the paper

“In this framework, the main job is done by the image synthesis network, a CycleGAN, that converts the labeled source images to synthesized target images.”

What the sources say

• Not checked yetcites Huo et al. (2018a)

In the paper

“We also reviewed data augmentation methods based on image synthesis, which generate images with larger appearance variability than those generated by the traditional data augmentation.”

In the paper

“The authors report a 9% improvement in Dice over the existing state of the art domain adaptation techniques.”

What the sources say

• Not checked yetcites Chen et al. (2019a)

In the paper

“Chen et al. (2019a) perform domain translation from the MR to CT domain for the task of heart CT segmentation using only MR image masks.”

What the sources say

• Not checked yetcites Chen et al. (2019a)

In the paper

“Chen et al. (2019b) makes use of MUNIT to translate between balanced steady-state free precession (bSSFP) images having masks for 3 cardiac structures and late-gadolinium enhanced (LGE) images that dont have any masks.”

What the sources say

• Not checked yetcites Chen et al. (2019b)

In the paper

“Harouni et al. (2018a) propose a modality independent model that is jointly trained using data from all modalities.”

What the sources say

• Not checked yetcites Harouni et al. (2018a)

In the paper

“The authors evaluate their approach on the five validation images provided by the challenge and show a 10% increase in Dice compared to a registration-based approach.”

What the sources say

• Not checked yetcites Chen et al. (2019b)

In the paper

“On average, they improve the Dice score by 9% over a baseline that performs segmentation in the CT domain.”

What the sources say

• Not checked yetcites Giger (2018) source

In the paper

“Chartsias et al. (2017) use CycleGAN to generate pairs of synthesized MR images and the corresponding myocardium masks from pairs of CT slices and their myocardium segmentation masks.”

What the sources say

• Not checked yetcites Chartsias et al. (2017)

In the paper

“The multi-dataset training scheme achieves 1.5% improvement in Dice over the state of the art single dataset approaches.”

What the sources say

• Not checked yetcites Dmitriev and Kaufman (2019)

In the paper

“Once the synthetic data is generated, the authors train a myocardium segmentation model using both synthetic MR and real MR images, demonstrating 10% improvement over the myocardium segmentation model trained using only the real MR images, when tested on 5 MR images.”

What the sources say

• Not checked yetcites Chartsias et al. (2017)

In the paper

“They observe that across different segmentation models, this augmentation significantly increases precision, recall, and Dice score.”

What the sources say

• Not checked yetcites Tang et al. (2019a)

In the paper

“They achieve 2% improvement in Dice over the baseline CycleGAN performance.”

What the sources say

• Not checked yetcites Chen et al. (2018)

In the paper

“With these new constraints, tested on 60 X-ray images, the suggested method improves the segmentation Dice by two or three points over using either one of them alone and over the vanilla CycleGAN.”

What the sources say

• Not checked yetcites Zhang et al. (2018c)

In the paper

“Huo et al. (2018a) suggest a joint image synthesis and segmentation framework that enables image segmentation for the target domain using unlabeled target images and labeled images from a source domain.”

What the sources say

• Not checked yetcites Huo et al. (2018a)

In the paper

“For the task of total intracranial volume segmentation, the Dice coefficient using domain adaptation is only 1% lower than the Dice coefficient of the model trained with the target labels (upper bound).”

What the sources say

• Not checked yetcites Huo et al. (2018b)

In the paper

“Pre-processing using this method resulted in significantly improved segmentation performance on both deep learning (HNN) and non deep learning (GrabCut) approaches on the DeepLesion dataset Yan et al. (2018).”

What the sources say

• Not checked yetcites Tang et al. (2018)
• Not checked yetcites Yan et al. (2018) source

In the paper

“Augmenting the dataset with 5000 synthesized images significantly improves all performance metrics.”

What the sources say

• Not checked yetcites Tang et al. (2019b)

In the paper

“Dmitriev and Kaufman (2019) train a multi-organ segmentation model using data from multiple single organ datasets.”

What the sources say

• Not checked yetcites Dmitriev and Kaufman (2019)

In the paper

“The authors show that synthesized images, when combined with traditional augmentation, achieve 4 points increase in Dice over the same model trained using only traditional data augmentation.”

What the sources say

• Not checked yetcites Abhishek and Hamarneh (2019)

In the paper

“Without typical data augmentation, the tumor segmentation model trained using both synthetic and real MR images achieves a significant performance gain over the model trained using only real MR images.”

What the sources say

• Not checked yetcites Shin et al. (2018)

In the paper

“The exception to this performance was seen for left ventricle segmentation, where a dedicated MRI model showed significantly higher performance.”

What the sources say

• Not checked yetcites Harouni et al. (2018a)

In the paper

“The authors evaluate this approach for melanoma segmentation in ISIC 2017 challenge dataset for Skin Lesion Analysis (1600 training and 400 test images) , demonstrating a 55% reduction in the annotation cost.”

What the sources say

• Not checked yetcites Gorriz et al. (2017)

In the paper

“Kuo et al. (2018) propose an active learning framework based on sample uncertainty and annotation cost.”

What the sources say

• Not checked yetcites Kuo et al. (2018)

In the paper

“Unlabeled medical images, although lack annotations, can still be used in conjunction with labeled data to train higher-performing segmentation models.”

In the paper

“The authors evaluate suggestive annotation on a histopathology dataset for gland segmentation (85 training and 80 test images) and a CT dataset for lymph node segmentation (37 training and 37 test images), achieving the full-dataset performance with only 50% of training data.”

What the sources say

• Not checked yetcites Yang et al. (2017)

In the paper

“The authors evaluate the suggested active learning method for intracranial hemorrhage segmentation on 1247 head CT scans (934 training/313 test), achieving the performance of a full-dataset model with 50% of the training set and 20% of annotation cost.”

What the sources say

• Not checked yetcites Kuo et al. (2018)

In the paper

“Sakinis et al. (2019) propose a semi-automated image segmentation method that enables a high quality segmentation with only a few user clicks.”

What the sources say

• Not checked yetcites Sakinis et al. (2019)

In the paper

“Recent methodologies overcome this limitation by estimating the reliability of pseudo annotations during training, by means of model consensus Min et al. (2018); Xia et al. (2020) or through the use of an evaluation network Nie et al. (2018).”

What the sources say

• Not checked yetcites Min et al. (2018)
• Not checked yetcites Xia et al. (2020)
• Not checked yetcites Nie et al. (2018)

In the paper

“The exception is the work of Kuo et al. (2018) where the authors included annotation cost in their sample selection method.”

What the sources say

• Not checked yetcites Kuo et al. (2018)

In the paper

“By ranking the examples using the fused metric metrics, the authors achieve similar performance to the model trained with the full dataset while using only 27% of the entire training set.”

What the sources say

• Not checked yetcites Ozdemir et al. (2018)

In the paper

“These results are significant because, unlike the previous works, the performance gains hold up when the models are trained using the full-size training datasets.”

What the sources say

• Not checked yetcites Zhou et al. (2019b)

In the paper

“When trained with only 0.5% of the training voxels, this scheme achieves the same performance as the model that is trained using the entire training set.”

What the sources say

• Not checked yetcites Sourati et al. (2019) source

In the paper

“Combined with an image synthesis network, the suggested method achieves the full-data performance with only 30-35% of annotated pixels.”

What the sources say

• Not checked yetcites Mahapatra et al. (2018)

In the paper

“The advantage of model pre-training using unlabeled medical data is that the learned knowledge is related to the target medical task; and thus, can be more effective than transfer learning from a foregin domain (e.g., Tajbakhsh et al. (2019a) and Ross et al. (2018)).”

What the sources say

• Not checked yetcites Tajbakhsh et al. (2019a)
• Not checked yetcites Ross et al. (2018)

In the paper

“Nearly all active learning works reviewed in this survey base sample selection on informativeness and diversity of samples, neglecting the cost associated with annotating a sample.”

In the paper

“However, the authors have not demonstrated a significant gain over the 2D and 3D performance baselines.”

What the sources say

• Not checked yetcites Angermann et al. (2019)

In the paper

“One caveat, however, is that these methods are not as effective when the training set grows in size”

In the paper

“Self-supervised learning offers several advantages: 1) demonstrated performance gains over the counterpart models trained from scratch”

In the paper

“However, one caveat is that the suggested method is evaluated only in a low data regime where the labeled and unlabeled training sets have images from 3 and 9 subjects, respectively.”

What the sources say

• Not checked yetcites Baur et al. (2017)

In the paper

“Informative representations can be particularly effective for 2D medical image segmentation”

In the paper

“One drawback of the above approaches is that the companion task (classification or detection) requires additional labels for each image.”

In the paper

“The suggested image representation enables a 4% gain in AUC compared with the existing state of the art trained using the standard Cartesian plane image representation.”

What the sources say

• Not checked yetcites Fu et al. (2018b)

In the paper

“In both scenarios, the suggested method improves Dice score by 1% over baselines that use majority voting for volume fusion.”

What the sources say

• Not checked yetcites Xia et al. (2018)

In the paper

“It is known that convolutional networks are inherently not rotation equivariant 1 .”

What the sources say

• Not checked yetcites Cohen and Welling (2016)

In the paper

“Simply adding this additional branch, which encouraged the model to learn a feature space relevant to diagnosis in addition to tissue segmentation, significantly improves the IoU of the vanilla network by 7%.”

What the sources say

• Not checked yetcites Mehta et al. (2018)

In the paper

“3) ease of use, because no or only minor architectural changes to the segmentation network is required.”

In the paper

“Image reconstruction can be considered an unsupervised task that still provides the regularization benefits of multi-task learning.”

In the paper

“however, they have not compared their method against a single view CNN approach.”

What the sources say

• Not checked yetcites Wang et al. (2017)

In the paper

“Testing on 393 nodules from the LIDC dataset, they achieve 7% increase in Dice over GrabCut”

What the sources say

• Not checked yetcites Wang et al. (2017)

In the paper

“The suggested methods often show only moderate performance gains over the counterparts trained using only labeled images.”

In the paper

“By jointly performing image reconstruction with segmentation, they observed improved segmentation performance in all types of brain tissue across different metrics.”

What the sources say

• Not checked yetcites Sun et al. (2018b)

In the paper

“report a 3.0% gain in Dice using U-net as segmentation network over the same network without star shape regularized loss.”

What the sources say

• Not checked yetcites Mirikharaji and Hamarneh (2018)

In the paper

“The recent semi-supervised learning frameworks have turned this weakness of regular convolutional networks into an opportunity to leverage unlabeled data.”

In the paper

“The studies outlined below demonstrate that adding a parallel task generally results in improved segmentation performance at little or no cost in terms of additional data.”

In the paper

“This is because model-generated annotations can be noisy, which has detrimental effects to the subsequent segmentation model.”

In the paper

“Semi-supervised learning method often use adversarial loss functions to bridge the dissimilarities between the representations of labeled and unlabeled images, thereby leveraging unlabeled data during the course of training.”

In the paper

“Noteworthy, the improved segmentation performance is mainly due to the contribution of handcrafted features rather than the multi-scale paradigm.”

What the sources say

• Not checked yetcites Ghafoorian et al. (2017) source

In the paper

“Their ablation studies show that multi task learning (segmentation and image-level classification) significantly increase segmentation when compared to the same network without the additional classification task.”

What the sources say

• Not checked yetcites Li et al. (2019a)

In the paper

“Mean teacher paradigm, a.k.a self-ensembling, is expected to outperform the previous paradigm where the segmentation network serves as both teacher and student.”

In the paper

“Models Genesis suggested by Zhou et al. (2019b) is a significant shift from this paradigm where a library of diverse self-supervised schemes, all formulated as an image restoration task, is used to generate self-supervision signal.”

What the sources say

• Not checked yetcites Zhou et al. (2019b)

In the paper

“The trend for tackling the problem of having only image-level labels is to use some form of class activation maps (CAMs), which can be binarized to generate a segmentation mask.”

In the paper

“recent research, as summarized below, has shown that it is possible to train fairly effective models with weak annotations.”

In the paper

“Since there is no information about which instances in the positive bag are actually positive instances, assigning the bag-level label to each instance in the bag would result in noisy and unreliable labels.”

In the paper

“And on BraTS 2015 Menze et al. (2014), consisting of 220 multi-modal MR images of brain tumors, 3D RNN-CRF slightly improves the Dice for tumor segmentation from 73.5% ± 10.5% to 73.8% ± 10.5% based upon a split of 85%/15% (187 training and 33 testing).”

What the sources say

• Not checked yetcites Monteiro et al. (2018)
• Not checked yetcites Menze et al. (2014) source

In the paper

“resulting in 18% to 45% reduction in Hausdorff distance without degrading other performance metrics such as Dice similarity coefficient.”

What the sources say

• Not checked yetcites Karimi and Salcudean (2019)

In the paper

“Their average IOU improvement across different tests over the baseline of just using CAM is 20%.”

What the sources say

• Not checked yetcites Izadyyazdanabadi et al. (2018)

In the paper

“Since the pixels (or voxels) are only partially labelled, the underlying theme in these methodologies is the use of a selective pixel loss wherein only the labeled pixels contribute to the loss.”

In the paper

“The authors use 26 labeled and 118 unlabeled abdominal CTA volumes for training and report 1.0% gain in Dice over the same architecture without anatomy priori on a test set of 26 volumes.”

What the sources say

• Not checked yetcites He et al. (2019)

In the paper

“Campanella et al. (2019) train a weakly supervised segmentation model for whole slide histopathology images using only slide-level labels.”

What the sources say

• Not checked yetcites Campanella et al. (2019) source

In the paper

“Combining both data augmentation and deep regularization, Ravishankar et al. (2017) report a 4.66% gain in Dice relative to a vanilla U-Net segmenting kidney on 2D ultrasound B-mode images (100 training and 131 testing).”

What the sources say

• Not checked yetcites Ravishankar et al. (2017)

In the paper

“The authors evaluate on 60 MR scans from WMH 2017 Challenge (36 training, 12 validation and 12 testing) and report between six to seven points increase in Dice over a baseline U-Net and other implementations of CRF.”

What the sources say

• Not checked yetcites Chen and de Bruijne (2018)

In the paper

“Nevertheless, CRFs are still the most commonly adopted and recognized approach to refine segmentation masks of both natural images (Schwing and Urtasun (2015), Chen et al. (2017) and Zheng et al. (2015)) and medical images (Roth et al. (2015), Chen and de Bruijne (2018) and Wachinger et al. (2018)).”

What the sources say

• Not checked yetcites Schwing and Urtasun (2015)
• Not checked yetcites Chen et al. (2017) source
• Not checked yetcites Zheng et al. (2015)

In the paper

“By using CRF for decision fusion, the authors report a 2.3% gain in Dice (73.8% ± 12.0% → 76.1% ± 8.7%) over a baseline without CRF on a MRI dataset consisting of 78 scans (52 for training and 26 for testing).”

What the sources say

• Not checked yetcites Cai et al. (2016)

In the paper

“Fu et al. (2016a) combine a multi-scale and multi-level CNN that has auxiliary output layers with a RNN-CRF, and achieve the state-of-the-art performance of vessel segmentation on three public fundus datasets (DRIVE, STARE and CHASE-DB1).”

What the sources say

• Not checked yetcites Fu et al. (2016a)

In the paper

“The suggested model achieves 1.2% and 2.0% improvement respectively in Dice over Ravishankar et al. (2017) for 3D endocardium and myocardium segmentation on a cine-MR dataset (1000 training and 200 testing).”

What the sources say

• Not checked yetcites Oktay et al. (2017) source
• Not checked yetcites Ravishankar et al. (2017)

In the paper

“On the PROMISE 2012 dataset, which consists of 50 3D-MRI prostate images, 3D RNN-CRF improves the Dice from 76.7% ± 10.9% to 78.0% ± 11.0% evaluated in a 5-fold cross-validation manner.”

What the sources say

• Not checked yetcites Monteiro et al. (2018)

In the paper

“However, the refinement stage is computationally expensive, taking 15-20 minutes per case with their multiple CPU implementation.”

What the sources say

• Not checked yetcites Duan et al. (2019) source

In the paper

“As a consequence, CRF is able to refine a collection of inaccurate and coarse pixel-level predictions, producing sharp boundaries and fine-grained segmentation masks.”

In the paper

“This form of data regularization results in a lower dimensional input space or an altered input space that facilitates the task of representation learning for the model particularly in the absence of large labeled datasets.”

In the paper

“Jia et al. (2017) use a multiple instance learning approach to generate pixel-level predictions given only image-level cancer labels for histopathology images.”

What the sources say

• Not checked yetcites Jia et al. (2017) source

In the paper

“However, they differ from the sparse label techniques in that the pixels that contribute to the loss are not predefined but decided on the fly.”

In the paper

“Zhang et al. (2019c) propose a method for brain extraction and brain tissue segmentation in 3D MR images with a sparse set of annotations, where only a fraction of the slices are annotated at irregular intervals.”

What the sources say

• Not checked yetcites Zhang et al. (2019c)

In the paper

“For the task of brain extraction, the suggested method trained with just 15% of the slices labeled yields results similar to that of a model trained with a 50% annotation rate.”

What the sources say

• Not checked yetcites Zhang et al. (2019c)

In the paper

“Of the weakly supervised approaches reviewed, the papers tackling sparse annotation have achieved the closest performance to their strongly supervised counterparts; however, the application of sparse annotations may not always be viable.”

In the paper

“They show a 7% increase in Dice when using this scheme, over an approach where all pixels contribute to the loss and undefined boundary regions are penalized.”

What the sources say

• Not checked yetcites Matuszewski and Sintorn (2018)

In the paper

“This method achieves 10% improvement in Dice over a model trained using the original diameter markings.”

What the sources say

• Not checked yetcites Cai et al. (2018)

In the paper

“Cai et al. (2018) train a 3D segmentation model using only 2D annotations, which consist of diameter markings along the short and long axes of each lesion on the slice where the lesion appears the largest.”

What the sources say

• Not checked yetcites Cai et al. (2018)

In the paper

“The latter, on the other hand, is more straightforward to implement, because it uses only the available labeled pixels with proper loss-weighting schemes.”

In the paper

“However, no comparison was made against a fully-supervised oracle or other state-of-the-art weakly supervised approaches.”

What the sources say

• Not checked yetcites Matuszewski and Sintorn (2018)

In the paper

“Of the weakly supervised approaches reviewed, the papers tackling sparse annotation have achieved the closest performance to their strongly supervised counterparts; however, the application of sparse annotations may not always be viable.”

In the paper

“However, when making use of multiple datasets, it is possible that the combined dataset now suffers from both scarce and weak annotations.”

In the paper

“Matuszewski and Sintorn (2018) perform virus segmentation in electron microscopy images given minimal manual annotation in the form of points or lines.”

What the sources say

• Not checked yetcites Matuszewski and Sintorn (2018)