Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation

In the paper

“Recently, a large body of research has studied the problem of medical image segmentation with imperfect datasets, tackling two major dataset limitations: scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations.”

In the paper

“The common limitations of medical image segmentation datasets include scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations.”

In the paper

“In response to this challenge, researchers from the medical imaging community have actively sought solutions, resulting in a diverse and effective set of techniques with demonstrated capabilities in handling scarce and weak annotations for the task of medical image segmentation.”

In the paper

“A recurring theme in many of the domain adaptation papers discussed in this section is the use of GANs, CycleGANs, or some sort of adversarial loss for the purpose of image reconstruction.”

In the paper

“In a low data regime where the training set has fewer than 50 MR images, self-supervised pre-training achieves a significant gain over random initialization. However, the performance gain becomes only marginal when the segmentation model is trained with 100 MR images.”

What the sources say

• Not checked yetcites Bai et al. (2019)

In the paper

“The authors show that this surrogate task is highly effective for diabetic retinopathy classification in fundus images and lung lobe segmentation in chest CT scans.”

What the sources say

• Not checked yetcites Tajbakhsh et al. (2019a)

In the paper

“When only 25 images are used for fine-tuning, the instrument segmentation model pre-trained via self-supervised learning achieves a Dice score of 0.61, which outperforms the counterpart model pre-trained using Microsoft COCO dataset and the model trained from scratch, both with a Dice score of 0.57.”

What the sources say

• Not checked yetcites Ross et al. (2018)

In the paper

“The segmentation model trained from the self-supervised model achieves a Dice score of 0.8, outperforming the model trained from scratch with a Dice of 0.72.”

What the sources say

• Not checked yetcites Spitzer et al. (2018)

In the paper

“The authors have evaluated Models Genesis for both image classification and segmentation in seven 2D and 3D medical datasets, demonstrating three and five points increase in IoU over 3D models trained from scratch for lung nodule segmentation and liver segmentation in CT scans, respectively.”

What the sources say

• Not checked yetcites Zhou et al. (2019b)

In the paper

“This approach is simple to implement; however, hindered by the quality of pseudo annotations, the resulting model achieves only a moderate level of improvement over the model trained using only labeled data.”

What the sources say

• Not checked yetcites Bai et al. (2017)

In the paper

“The results show that the suggested semi-supervised learning framework based on pseudo-labeling and the two-stream network is noise-resilient, significantly outperforming fully supervised models under varying levels of label noise present in labeled images.”

What the sources say

• Not checked yetcites Min et al. (2018)

In the paper

“The authors show that the suggested framework is highly effective for myocardial segmentation in low-data regime, but the performance gap closes as the size of training set increases.”

What the sources say

• Not checked yetcites Chartsias et al. (2018)

In the paper

“While image colorization proved more effective than random initialization in training with both small ( ≈ 140 images) and large ( ≈ 1400 images) training sets, it was outperformed by transfer learning from ImageNet, presumably because the distance between tele-medicine skin images and ImageNet is small.”

What the sources say

• Not checked yetcites Tajbakhsh et al. (2019a)

In the paper

“Performance evaluation on multiple datasets show that both methods above are effective in boosting segmentation performance in low data regime; however, the performance gains tend to be limited in the presence of large labeled training sets.”

What the sources say

• Not checked yetcites Li et al. (2019b)
• Not checked yetcites Bortsova et al. (2019)

In the paper

“The suggested model however yields mixed results, yielding comparable Dice to a supervised model on the larger test set (60 images) while achieving 1 point increase in Dice on the smaller test set (20 images).”

What the sources say

• Not checked yetcites Zhang et al. (2017b)

In the paper

“Test results on MR images from 30 subjects demonstrate that the semi-supervised model achieves three points increase in Dice over a supervised model, further outperforming semi-supervised methods by Zhang et al. (2017b).”

What the sources say

• Not checked yetcites Cui et al. (2019)
• Not checked yetcites Zhang et al. (2017b)

In the paper

“Zhang et al. (2019a) show that data augmentation by adjusting image quality enables the largest performance gain in MR images, with largest improvement coming from image sharpening through the application of unsharp masking.”

What the sources say

• Not checked yetcites Zhang et al. (2019a)

In the paper

“The data augmentation methods used in medical imaging can be grouped by the image property they intend to manipulate Zhang et al. (2019a).”

What the sources say

• Not checked yetcites Zhang et al. (2019a)

In the paper

“For example, traditional data augmentation is handicapped by the large correlation between the original training set and the augmented examples.”

In the paper

“Transfer learning from natural images only benefits 2D medical image segmentation models, with no benefits to the common 3D medical image segmentation models.”

In the paper

“Our focus is motivated by the fact that image segmentation requires the strongest supervision among other vision tasks such as classification and detection; and thus, most vulnerable to the quality and quantity of annotations.”

In the paper

“The proactive approaches, on the other hand, actively enlarge the training set through cost-effective annotation and synthetic data generation or change the training paradigm by leveraging unlabeled data and using additional model regularization during training.”

In the paper

“These common image properties consists of image quality, image appearance, and image layout.”

In the paper

“Manifold mixup has recently proved effective for prostate cancer segmentation on MR image Jung et al. (2019), improving Dice by two to four points depending on the neural architecture used for segmentation.”

What the sources say

• Not checked yetcites Jung et al. (2019)

In the paper

“In the presence of these dataset shortcomings, even the most advanced segmentation models may fail to generalize to datasets from real-world clinical settings.”

In the paper

“Traditional data augmentation has proved effective in reducing over-fitting and improving test performance for both natural and medical images Zhang et al. (2016).”

What the sources say

• Not checked yetcites Zhang et al. (2016)

In the paper

“However, Zhang et al. (2018d) demonstrate that the above offline data augmentation may only be partially effective and in some cases can even deteriorate the performance.”

What the sources say

• Not checked yetcites Zhang et al. (2018d)

In the paper

“For the task of brain structure segmentation, the suggested data augmentation method enables four points increase in Dice over a model trained using traditional data augmentation and 3 points increase in Dice over atlas-based data augmentation.”

What the sources say

• Not checked yetcites Zhao et al. (2019a)

In the paper

“Using 16 subjects from each modality for training and 4 for testing, they achieve substantial performance boost over single domain training and 2% improvement in Dice over other domain adaptation techniques.”

What the sources say

• Not checked yetcites Dou et al. (2018)

In the paper

“Their results showed that the baseline was actually at par with or in some cases outperformed most variants, the only exception being the fourth variant, which consistently outperformed the baseline and other dual stream variants.”

What the sources say

• Not checked yetcites Valindria et al. (2018) source

In the paper

“This trend is in part due to the 3D nature of medical images, which hampers transfer learning from 2D models trained on natural images, and also partially due to the promising performance of shallower segmentation networks in medical imaging, which unlike deep models may not benefit from fine-tuning.”

In the paper

“The intuition behind this joint optimization is that the training process can benefit from the complementary information between the synthesis and segmentation networks.”

In the paper

“As individual datasets tend to be small and typically originate from a single institution, they are inherently biased and the resulting models tend to perform poorly in the real world.”

In the paper

“Dataset fusion techniques leverage multiple datasets to train a universal segmentation model based on heterogeneous, disjoint datasets, offering two advantages: 1) more efficient training, as multiple models are consolidated into a single model, and 2) enhanced regularization, as data from multiple sources can provide further supervision.”

In the paper

“Giger (2018) propose converting the CT (target) domain to the MR (source) domain and then using an existing atlas-based algorithm (MALP-EM) to perform the segmentation on the converted MR images.”

What the sources say

• Not checked yetcites Giger (2018) source

In the paper

“They improve the Dice score on CT images by eight points and MR images by two points over other methods that use both real and synthetic data for training.”

What the sources say

• Not checked yetcites Zhang et al. (2018d)

In the paper

“For this purpose, the authors first convert annotated CT scans to digitally reconstructed radiographs (DRRs) via a 3D to 2D projection, and then learn a mapping between 815 DRRs and 73 training X-ray images.”

What the sources say

• Not checked yetcites Zhang et al. (2018c)

In the paper

“While CycleGAN provides a one-is-to-one mapping between the two domains, MUNIT allows a one-is-to-many mapping by sampling from the style encoding distribution.”

What the sources say

• Not checked yetcites Zhu et al. (2017)
• Not checked yetcites Huang et al. (2018a)

In the paper

“Chen et al. (2018) use the CycleGAN with an additional semantic adversarial loss, which is used to distinguish between source segmentation masks and segmentation predictions of the converted target to source images.”

What the sources say

• Not checked yetcites Chen et al. (2018)

In the paper

“The authors further extend their work in Huo et al. (2018b) for the task of splenomegaly and total intracranial volume segmentation, reporting 2% improvement in Dice over the existing state of the art-a 2-stage CycleGAN followed by a separate segmentation network.”

What the sources say

• Not checked yetcites Huo et al. (2018b)

In the paper

“If the segmentation masks are available for both domains, there is no longer a distinction between the choice of source and target domains.”

In the paper

“Although these methods are more effective for handling data scarcity as well as rare conditions, they are more demanding to implement, because their training schemes typically require adversarial networks and additional labeled or unlabeled datasets.”

In the paper

“Transfer learning from natural images has been widely adopted for medical image classification Tajbakhsh et al. (2016); Shin et al. (2016); however the application to medical image segmentation has been scarce.”

What the sources say

• Not checked yetcites Tajbakhsh et al. (2016) source
• Not checked yetcites Shin et al. (2016) source

In the paper

“Active learning is a cost-effective approach to enlarge the training datasets; and thus, it is highly amenable to the problem of limited annotation budget in medical image segmentation where clinical experts have limited availability, annotation cost is high, and the amount of unlabelled data is usually non-trivial.”

In the paper

“however, the appealing feature of domain adaptation methods without target annotations is independence from the target domain annotations, which makes these methods the only viable solution to deal with unlabeled target domain images.”

In the paper

“The key idea consists of pre-training the model using unlabeled medical data, which is easier to obtain, and then fine-tune the pre-trained model for the target medical vision task using the limited labeled data available for training.”

In the paper

“The results show that the suggested interactive segmentation method outperforms traditional interactive segmentation methods in both accuracy and speed of annotation.”

What the sources say

• Not checked yetcites Wang et al. (2018a)

In the paper

“Wang et al. (2018b) proposes a framework consisting of a proposal network and a refinement network where the former generates a base segmentation mask whereas the latter refines the base mask according to the suggestions provided by the user.”

What the sources say

• Not checked yetcites Wang et al. (2018b)

In the paper

“Reviewing the literature, we have identified two types of surrogate tasks: 1) image-to-scalar where an encoder network is pre-trained for a surrogate image classification or regression task; 2) image-to-image where an encoder-decoder network is pre-trained for a surrogate image regression task such as image colorization or image denoising.”

In the paper

“A limitation with the previous approaches is that the images that have expert annotation and images with pseudo segmentation masks are treated equally during training. As such, errors in the pseudo labels can degrade the quality of the resulting models.”

In the paper

“Semi-supervised learning with pseudo annotations has shown promising performance, producing models that outperform the counterparts trained using only labeled data.”

In the paper

“Sun et al. (2018a) propose an interactive method for segmenting fuzzy boundaries, wherein the user first places a point roughly at the center of the object, and then the model performs object delineation for the user-specified structure.”

What the sources say

• Not checked yetcites Sun et al. (2018a)

In the paper

“Active learning, in its general form, requires the availability of a base segmentation model; thus, a minimal set of base annotations is necessary.”

In the paper

“While the former approach seems particularly suitable for a downstream image classification task, it can still be used to initialize the encoder of a segmentation network, in which case the decoder should be initialized with random weights.”

In the paper

“In general, the models trained with target domain annotations are bound to generalize better to the target domain”

In the paper

“While active learning methods are typically iterative and slow, recent works Mahapatra et al. (2018); Zheng et al. (2019) present a 1-shot, fast approach to active learning with comparable performance to the iterative counterparts, making active learning an even more attractive methodology.”

What the sources say

• Not checked yetcites Mahapatra et al. (2018)
• Not checked yetcites Zheng et al. (2019)

In the paper

“Interactive segmentation can accelerate the annotation process by allowing the expert annotators to interactively correct an initial segmentation mask generated by a model.”

In the paper

“The majority of domain adaptation techniques require some form of adversarial training, making them tricky to train.”

In the paper

“One limitation of this work, however, is that computational complexity is super quadratic with respect to the number of parameters, because the Fisher matrix has as many rows and columns as the number of parameters in the network.”

What the sources say

• Not checked yetcites Sourati et al. (2018)

In the paper

“We have identified three scenarios wherein unlabeled medical images have aided medical image segmentation: 1) self-supervised pre-training where unlabeled images are used to pre-train a segmentation network; 2) semi-supervised learning with pseudo labels where unlabeled images are labeled by a segmentation model and then used as new examples during training; and 3) semi-supervised learning without”

In the paper

“On the other hand, transfer learning and dataset fusion require minimal changes to the network architecture, allowing for simple and successful joint dataset training.”

In the paper

“Yang et al. (2017) propose a framework called suggestive annotation where the candidate samples for each round of annotation are selected through a 2-stage screening process.”

What the sources say

• Not checked yetcites Yang et al. (2017)

In the paper

“These results are comparable to Yang et al. (2017), but are obtained using datasets that are two orders of magnitude larger in size.”

What the sources say

• Not checked yetcites Kuo et al. (2018)
• Not checked yetcites Yang et al. (2017)

In the paper

“To put this in perspective, with only 1 user click, this semi-automated method can achieve a Dice of 0.64 for segmenting an unseen structure, colon cancer, in 126 Abdomen CT scans, outperforming the best automated model with a Dice of 0.56.”

• Also stated in: § 4.3.3. Summary · p. 13 · plain

  “In particular, the work by Sakinis et al. (2019) was quite effective, where 1 user click led to a Dice of 0.64 for segmenting colon cancer, outperforming the best automated model with a Dice of 0.56.”

What the sources say

• Not checked yetcites Sakinis et al. (2019)

In the paper

“Interactive segmentation complements active learning in achieving cost-effective annotation: the latter identifies which images to be annotated whereas the former reduces the time required to complete the annotation of a selected image.”

In the paper

“Nevertheless, these semi-supervised schemes are effective in coping with limited labeled training sets, improving the Dice score by a couple of points in most cases.”

In the paper

“This approach outperforms other U-net variants by several points and the popular Mask R-CNN algorithm by 20 points when the Dice score is used for performance evaluation.”

What the sources say

• Not checked yetcites Huang et al. (2018b)

In the paper

“Since these methods accommodate the segmentation task for the labeled data and an unsupervised task for the unlabeled data, they typically consist of complex neural architectures.”

In the paper

“Altered image representations consist of projecting or transforming the images into a more informative or compact representation, which present deep models with an easier problem to solve, thereby reducing the need for large training sets.”

In the paper

“On a test set of 46 cases, their multi-scale architecture equipped with the hand-crafted features achieves 6% increase in Dice over their single scale baseline that does not incorporate the handcrafted features.”

What the sources say

• Not checked yetcites Ghafoorian et al. (2017) source

In the paper

“The latter enables training the model in the source domain using both labeled and unlabeled data, increasing the effective size of the training set, mitigating the problem of limited annotations in the source domain.”

In the paper

“This limitations has been recently addressed in Min et al. (2018); Nie et al. (2018), but at the expense of relatively complex segmentation architectures.”

What the sources say

• Not checked yetcites Min et al. (2018)
• Not checked yetcites Nie et al. (2018)

In the paper

“given the sheer number of potential conditions, a single model per condition is not a scalable approach for clinical use.”

What the sources say

• Not checked yetcites Harouni et al. (2018b)

In the paper

“The suggested method outperforms other modifications such as CRF post-processing as well as sophisticated test-time data augmentation, winning the first place in the 2018 BraTS challenge”

What the sources say

• Not checked yetcites Myronenko (2018)
• Not checked yetcites Menze et al. (2014) source

In the paper

“The suggested framework is scalable to a large library of surrogate tasks, because all tasks in the library can share the same encoder and decoder during training, eliminating the need for task-specific decoders.”

What the sources say

• Not checked yetcites Zhou et al. (2019b)

In the paper

“2) ease of implementation owing to the intuitive underlying proxy tasks”

In the paper

“Compared to the previous strategy, these methods do not attempt to generate annotations for the unlabeled images; as such, they are not vulnerable to annotation noise.”

In the paper

“While this model does not exceed the state of the art performance for any one given modality or target, performance is competitive across all domains.”

What the sources say

• Not checked yetcites Harouni et al. (2018b)

In the paper

“The authors also report that the presence of the image classification branch results in slightly improved segmentation performance.”

What the sources say

• Not checked yetcites Harouni et al. (2018b)

In the paper

“whereas compact representations can benefit 3D applications where the curse of dimensionality requires large annotated datasets.”

In the paper

“The authors show that the local (rather than global) decoding approach preserves spatial details as well as decreases GPU footprint.”

What the sources say

• Not checked yetcites Huang et al. (2018b)

In the paper

“By imposing this additional regularization term, not only does the model behave more predictably, but also it can utilize the unlabeled data available for training.”

In the paper

“report an 8% gain in Dice and a 10% gain in Hausdorff score over a baseline that uses generalized Dice as the loss function on ISLES dataset compared to marginal improvement on the WMH dataset.”

What the sources say

• Not checked yetcites Kervadec et al. (2019)

In the paper

“While both approaches improve the model performance over their respective baselines, it would be interesting to see how these two techniques can be combined together to achieve even higher gains.”

In the paper

“We then reviewed multi-task learning for medical image segmentation, which leads to consistent improvement over the single-task segmentation models and further enables a consolidated framework for multi-modality multi-condition medical image segmentation.”

In the paper

“The authors attribute the inferior performance by Ravishankar et al. (2017) to over-regularization, which they have overcome by replacing CDAE with a frozen encoder during training.”

What the sources say

• Not checked yetcites Oktay et al. (2017) source
• Not checked yetcites Ravishankar et al. (2017)

In the paper

“While both locallyand fully-connected CRF have proved effective in 2D segmentation applications, their extension to 3D applications has shown only mixed results.”

In the paper

“The work by Kamnitsas et al. (2017) is the first to extend FC-CRF to 3D brain lesion segmentation in MR Images”

What the sources say

• Not checked yetcites Kamnitsas et al. (2017) source

In the paper

“the authors evaluate on 82 contrast-enhanced abdominal CT volumes in a 4-fold cross-validation manner, and report an average of 3.3% gain in Dice using CRF that falls short of a 6.9% gain using Gaussian smoothing.”

What the sources say

• Not checked yetcites Roth et al. (2015)

In the paper

“Finally, we reviewed RNN-CRF, which operates on CNN feature maps and is trained end-to-end along with the segmentation model, thereby addressing the limitations of parameter tuning and susceptibility to image noise.”

In the paper

“Their approach outperforms a naive partial supervision implementation, which does not benefit from shape prior regularization, by 1.69% in Dice with ResNet-101 as backbone for 2D task and by 0.45% in Dice using 3D U-Net as the backbone models, both based on 5-fold cross-validation.”

What the sources say

• Not checked yetcites Zhou et al. (2019a)

In the paper

“Weakly supervised techniques can take advantage of bounding boxes or image level labels.”

In the paper

“The papers reviewed in this section address the problem of image segmentation with the weakest form of annotation: image-level labels.”

In the paper

“the authors report performance superior to 3D-ACNN Oktay et al. (2017), achieving 4.3% increase in Dice and 4.15mm (40% less) improvement in Hausdorff distance .”

What the sources say

• Not checked yetcites Duan et al. (2019) source
• Not checked yetcites Oktay et al. (2017) source

In the paper

“For the task of brain structure segmentation, the suggested method is evaluated on 9000 MRI scans, achieving Dice scores ranging between 0.50 and 0.80 without any comparison against supervised methods.”

What the sources say

• Not checked yetcites Dalca et al. (2018)

In the paper

“By learning to classify the image as a whole, MIL learns to predict instancelevel predictions, or equivalently the segmentation prediction for the input image.”

In the paper

“the authors note that configuring 3D FC-CRF is a laborious task.”

What the sources say

• Not checked yetcites Kamnitsas et al. (2017) source

In the paper

“These methods further require extensive parameter tuning and may be susceptible to image noise as they operate directly on pixel information.”

In the paper

“RNN-CRF organically integrates CRF with CNNs, making it possible to train the whole network in an end-to-end manner.”

In the paper

“the 2.5D FCN trained on the LR cases with refinement achieves comparable performance relative to the 2.5D FCN trained on the HR cases, trailing by 2.0% for most regions in Dice and 0.5mm worse in Hausdorff distance.”

What the sources say

• Not checked yetcites Duan et al. (2019) source

In the paper

“Compared to shallow approaches, deep regularization, a.k.a deep supervision or deep priors, is less prone to image noise and more semantically and structurally aware.”

In the paper

“Variants of auto-encoders have recently been adopted for automatic error correction in medical images Larrazabal et al. (2019); Tajbakhsh et al. (2019b), outperforming conditional random fields (CRFs) especially in the presence of domain shift and limited annotations.”

What the sources say

• Not checked yetcites Larrazabal et al. (2019)
• Not checked yetcites Tajbakhsh et al. (2019b)

In the paper

“Also, except for multi-task learning which requires additional annotations, input space regularization and shape regularization require no further annotations; and thus, can be taken advantage of at the cost of changing the data pipeline and minor architectural changes, respectively.”

In the paper

“their 3D generalization on BraTS dataset Menze et al. (2014) (274 training and 110 testing) leads to marginal performance gains in Dice, i.e. 3.7% over Random Forests, 0.3% over an ensemble method, and merely 0.7% over their proposed architecture”

What the sources say

• Not checked yetcites Kamnitsas et al. (2017) source
• Not checked yetcites Menze et al. (2014) source

In the paper

“The authors acknowledge that the improvements by 3D RNN-CRF are inconclusive”

What the sources say

• Not checked yetcites Monteiro et al. (2018)

In the paper

“FC-CRF has proven effective as a segmentation post-processing solution for both natural images Chen et al. (2017) and 2D medical images, e.g. Fu et al. (2016b) in retinal images, Gao et al. (2016) on individual CT slices.”

What the sources say

• Not checked yetcites Chen et al. (2017) source
• Not checked yetcites Fu et al. (2016b)
• Not checked yetcites Gao et al. (2016)