DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation

In the paper

“Machine learning methods [6], [37], [38] have been proposed to learn from user interactions. They can achieve higher segmentation accuracy with fewer user interactions. However, they are limited by hand-crafted features that depend on the user's experience.”

What the sources say

• Not checked yetcites [6] Wang G., Zuluaga M. A., Pratt R., Aertsen M., Doel T., Klusmann M., David A. L., Deprest J., Vercauteren T., and Ourselin S., “ Slic-Seg: A minimally interactive segmentation of the placenta from sparse and motion-corrupted fetal MRI in multiple views,” Med. Image Anal., vol. 34, pp. 137– 147, 2016. 27179367 PMC5052128 source
• Not checked yetcites [37] Barinova O., Shapovalov R., Sudakov S., and Velizhev A., “ Online random forest for interactive image segmentation,” in Proc. Int. Workshop Exp. Economics Mach. Learn., 2012, pp. 1– 8.
• Partly supported

  The Introduction describes machine-learning segmentation that learns from user annotations and requires discriminative features. Model training reports segmenting a biological volume with just two brief annotations, while Data Pre-processing and the Discussion say users must choose appropriate features and parameters. The paper does not establish that these methods achieve higher segmentation accuracy with fewer interactions, or explicitly identify dependence on user experience as a general limitation. (LLM, conf=0.82)

  cites [38] Luengo I., Darrow M. C., Spink M. C., Sun Y., Dai W., He C. Y., Chiu W., Pridmore T., Ashton A. W., Duke E. M., Basham M., and French A. P., “ SuRVoS: Super-region volume segmentation workbench,” J. Structural Biol., vol. 198, no. 1, pp. 43– 53, 2017. PMC5405849 28246039 sourceExperimental — accuracy not yet measured

In the paper

“To overcome the drawbacks of successive max-pooling and downsampling that lead to a loss of feature map resolution, dilated convolution [22], [23] was proposed to preserve the resolution of feature maps and enlarge the receptive field to incorporate larger contextual information.”

What the sources say

• Supported

  Section 3.1 (pp. 4–5) says atrous convolution enables denser, higher-resolution feature maps despite downsampling and enlarges filters’ field of view to assimilate more context. The Introduction (p. 1) also states both points together. (LLM, conf=0.94)

  cites [22] Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “ Semantic image segmentation with deep convolutional nets and fully connected CRFs,” arXiv preprint arXiv:1412.7062. 28463186 source
• Supported

  The Introduction (p. 2) says the dilated-convolution module aggregates multi-scale contextual information without losing resolution. Section 2 (p. 2) states that dilated convolutions expand the receptive field without losing resolution or coverage. (LLM, conf=0.94)

  cites [23] Yu F. and Koltun V., “ Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122. source

In the paper

“However, most of these methods rely on low-level features and require a relatively large amount of user interactions to deal with images with low contrast and ambiguous boundaries.”

What the sources say

• Not checked yetcites [8] Boykov Y. Y. and Jolly M. P., “ Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images,” in Proc. 8th IEEE Int. Conf. Comput. Vis., 2001, pp. 105– 112. source
• Partly supported

  The Conclusions (page not provided) say the paper’s own method relies mainly on pixel-value distributions and that overlapping distributions may take enough user interactions to become tedious. The Introduction also describes ambiguous images as a reason for user intervention. These passages do not establish that most interactive segmentation methods require relatively many interactions specifically for low-contrast images or ambiguous boundaries. (LLM, conf=0.83)

  cites [35] Bai X. and Sapiro G., “ A geodesic framework for fast interactive image and video segmentation and matting,” in Proc. IEEE 11th Int. Conf. Comput. Vis., 2007, pp. 1– 8. sourceExperimental — accuracy not yet measured
• Not checked yetcites [7] Grady L., “ Random walks for image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 11, pp. 1768– 1783, Nov. 2006. 17063682 source

In the paper

“obtains comparable and even higher accuracy with fewer user interventions and less time compared with traditional interactive methods.”

In the paper

“HighRes3DNet and 3D P-Net(b5) achieve similar performance, and they are better than DeepMedic.”

In the paper

“DeepIGeoS requires far fewer user interactions and less user time.”

In the paper

“In addition, CRF-Net(f) is better than CRF-Net(g).”

In the paper

“the proposed 3D P-Net has far fewer parameters compared with HighRes3DNet.”

In the paper

“DeepIGeoS only needs two short strokes to get an accurate segmentation, while the other methods require far more scribbles to get similar results.”

In the paper

“Significant improvement from 2D R-Net”

In the paper

“3D P-Net outperforms these three counterparts with 86.68”

In the paper

“Fig. 8 shows 2D R-Net with geodesic distance performs better than min-cut user-editing and 2D R-Net(Euc) that uses euclidean distance.”

In the paper

“our proposed method achieves better results than automatic CNNs.”

In the paper

“Significant improvement from 3D R-Net”

In the paper

“ASSD of 1.43”

In the paper

“It requires far less user time compared with traditional interactive methods”

In the paper

“Dice score of 89.93”

In the paper

“However, the euclidean distance does not take advantage of image context information.”

• Also stated in: § User Interaction-Based Geodesic Distance Maps · plain

  “However, the euclidean distance treats each direction equally and does not take the image context into account.”

In the paper

“In [40], a deep interactive object selection method was proposed where user-provided clicks are transformed into euclidean distance maps and then concatenated with the input of FCNs.”

What the sources say

• Not checked yetcites [40] Xu N., Price B., Cohen S., Yang J., and Huang T., “ Deep interactive object selection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 373– 381. source

In the paper

“Third, it is efficient and can respond to user interactions in real time, which leads to very short user time.”

In the paper

“DeepCut [32] employs user-provided bounding boxes as annotations to train CNNs for the segmentation of fetal MRI.”

What the sources say

• Not checked yetcites [32] Rajchl M., Lee M., Oktay O., Kamnitsas K., Passerat-Palmbach J., Bai W., Rutherford M., Hajnal J., Kainz B., and Rueckert D., “ DeepCut: Object segmentation from bounding box annotations using convolutional neural networks,” IEEE Trans. Medical Imaging, vol. 36, no. 2, pp. 674– 683, Feb. 2017. 27845654 PMC7115996 source

In the paper

“Recently, deep learning with convolutional neural networks (CNNs) has achieved the state-of-the-art performance in many image analysis applications [10].”

What the sources say

• Not checked yetcites [10] Litjens G., Kooi T., Bejnordi B. E., Setio A. A. A., Ciompi F., Ghafoorian M., van der Laak J. A., Ginneken B. V., and Sánchez C. I., “ A survey on deep learning in medical image analysis,” Med. Image Anal., vol. 42, pp. 60– 88, 2017. 28778026 source

In the paper

“SlicSeg [6] accepts user-provided scribbles in a single start slice to train an Online Random Forest for 3D segmentation, but lacks in flexibility to allow further user-editing.”

What the sources say

• Not checked yetcites [6] Wang G., Zuluaga M. A., Pratt R., Aertsen M., Doel T., Klusmann M., David A. L., Deprest J., Vercauteren T., and Ourselin S., “ Slic-Seg: A minimally interactive segmentation of the placenta from sparse and motion-corrupted fetal MRI in multiple views,” Med. Image Anal., vol. 34, pp. 137– 147, 2016. 27179367 PMC5052128 source

In the paper

“For example, the widely used ITK-SNAP [5] takes user-provided seed pixels or blobs as a starting point and employs an active contour model for segmentation. It requires most of the user interactions to be given at the beginning and the underpinning model can hardly be refined with additional user interactions once an initial segmentation is obtained.”

What the sources say

• Not checked yetcites [5] Yushkevich P. A., Piven J., Hazlett H. C., Smith R. G., Ho S., Gee J. C., and Gerig G., “ User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability,” NeuroImage, vol. 31, no. 3, pp. 1116– 1128, 2006. 16545965 source

In the paper

“Typical CNNs [16], [17], [18] were originally designed for image classification tasks.”

What the sources say

• Not checked yetcites [16] Krizhevsky A., Sutskever I., and Hinton G. E., “ ImageNet classification with deep convolutional neural networks,” in Proc. 25th Int. Conf. Neural Inf. Process. Syst., 2012, pp. 1097– 1105.
• Not checked yetcites [17] Simonyan K. and Zisserman A., “ Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556. source
• Not checked yetcites [18] He K., Zhang X., Ren S., and Sun J., “ Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770– 778. source

In the paper

“Some early works adapted such networks for pixel labeling with patch or region-based methods [19], [20]. Such methods achieved higher accuracy than traditional methods that relied on hand-crafted features. However, they suffered from inefficiency for testing.”

What the sources say

• Not checked yetcites [19] Havaei M., Davy A., Warde-Farley D., Biard A., Courville A., Bengio Y., Pal C., Jodoin P.-M., and Larochelle H., “ Brain tumor segmentation with deep neural networks,” Med. Image Anal., vol. 35, pp. 18– 31, 2016. 27310171 source
• Not checked yetcites [20] Girshick R., Donahue J., Darrell T., and Malik J., “ Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 580– 587. source

In the paper

“GrabCut [9] works with a user-provided bounding box to start the segmentation and requires fewer interactions compared with Graph Cuts [8], but the performance is still limited by the representativity of the underpinning GMM.”

What the sources say

• Not checked yetcites [9] Rother C., Kolmogorov V., and Blake A., “ GrabCut: Interactive foreground extraction using iterated graph cuts,” ACM Trans. Graph., vol. 23, no. 3, pp. 309– 314, 2004. source
• Not checked yetcites [8] Boykov Y. Y. and Jolly M. P., “ Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images,” in Proc. 8th IEEE Int. Conf. Comput. Vis., 2001, pp. 105– 112. source

In the paper

“ScribbleSup [39] also trains CNNs for semantic segmentation supervised by scribbles.”

What the sources say

• Not checked yetcites [39] Lin D., Dai J., Jia J., He K., and Sun J., “ ScribbleSup: Scribble-supervised convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3159– 3167. source

In the paper

“FCNs [11] take an entire image as input and give a dense segmentation. In order to overcome the problem of loss of spatial resolution due to multi-stage max-pooling and downsampling, it uses a stack of deconvolution (a.k.a. upsampling) layers and activation functions to upsample the feature maps.”

What the sources say

• Not checked yetcites [11] Long J., Shelhamer E., and Darrell T., “ Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 3431– 3440. 27244717

In the paper

“However, they have not yet been adapted to interactive medical image segmentation.”

In the paper

“Multi-scale features extracted from CNNs have been shown to be effective for improving segmentation accuracy [11], [22], [23].”

What the sources say

• Not checked yetcites [11] Long J., Shelhamer E., and Darrell T., “ Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 3431– 3440. 27244717 source
• Not checked yetcites [22] Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “ Semantic image segmentation with deep convolutional nets and fully connected CRFs,” arXiv preprint arXiv:1412.7062. 28463186 source
• Not checked yetcites [23] Yu F. and Koltun V., “ Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122. source

In the paper

“the geodesic distance transform [35], [36], [41] encodes spatial regularization and contrast-sensitivity”

What the sources say

• Not checked yetcites [35] Bai X. and Sapiro G., “ A geodesic framework for fast interactive image and video segmentation and matting,” in Proc. IEEE 11th Int. Conf. Comput. Vis., 2007, pp. 1– 8. source
• Not checked yetcites [36] Criminisi A., Sharp T., and Blake A., “ GeoS: Geodesic image segmentation,” in Proc. Eur. Conf. Comput. Vis., 2008, pp. 99– 112. source
• Not checked yetcites [41] Kontschieder P., Kohli P., Shotton J., and Criminisi A., “ GeoF geodesic forests for learning coupled predictors,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 65– 72. source

In the paper

“Random Walks [7] and Graph Cuts [8] learn from scribbles and allow the user to provide additional scribbles for refinement. They use Random Walker and Gaussian Mixture Model (GMM) as the underpinning model. However, they need a large number of scribbles to get satisfactory segmentation.”

What the sources say

• Not checked yetcites [7] Grady L., “ Random walks for image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 11, pp. 1768– 1783, Nov. 2006. 17063682 source
• Not checked yetcites [8] Boykov Y. Y. and Jolly M. P., “ Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images,” in Proc. 8th IEEE Int. Conf. Comput. Vis., 2001, pp. 105– 112. source

In the paper

“Graphical models such as CRFs [22], [42], [43] have been widely used to enhance segmentation accuracy by introducing spatial consistency.”

What the sources say

• Not checked yetcites [22] Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “ Semantic image segmentation with deep convolutional nets and fully connected CRFs,” arXiv preprint arXiv:1412.7062. 28463186 source
• Not checked yetcites [42] Boykov Y. and Kolmogorov V., “ An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 9, pp. 1124– 1137, Sep. 2004. 15742889 source
• Not checked yetcites [43] Yuan J., Bae E., and Tai X. C., “ A study on continuous max-flow and min-cut approaches,” in Proc. IEEE Comput. Society Conf. Comput. Vis. Pattern Recognit., 2010, pp. 2217– 2224. source

In the paper

“Despite the large number of existing interactive segmentation methods [3], most of them are confronted by requiring a large amount of user interactions and long user time, or limited learning ability with their underpinning models.”

What the sources say

• Not checked yetcites [3] Zhao F. and Xie X., “ An overview of interactive medical image segmentation,” Ann. BMVA, vol. 2013, no. 7, pp. 1– 22, 2013.

In the paper

“For instance, 3D U-Net [21] learns from sparsely annotated images and can be used for semi-automatic segmentation.”

What the sources say

• Not checked yetcites [21] Abdulkadir A., Lienkamp S. S., Brox T., and Ronneberger O., “ 3D U-Net: Learning dense volumetric segmentation from sparse annotation,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2016, pp. 424– 432. source

In the paper

“Alternatively, mean-field approximation [45], [50], [52] is often used for efficient inference of the CRF while allowing learning parameters by back-propagation.”

What the sources say

• Not checked yetcites [45] Krähenbühl P. and Koltun V., “ Efficient inference in fully connected CRFs with gaussian edge potentials,” in Proc. Neural Inf. Process. Syst. Conf., 2011, pp. 109– 117. source
• Not checked yetcites [50] Zheng S., Jayasumana S., Romera-Paredes B., Vineet V., Su Z., Du D., Huang C., and Torr P. H. S., “ Conditional random fields as recurrent neural networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529– 1537. source
• Not checked yetcites [52] Vemulapalli R., Tuzel O., Liu M.-y., and Chellappa R., “ Gaussian conditional random field network for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3224– 3233. source

In the paper

“in which the same initial segmentation and the same set of user interactions were used by the five refinement methods.”

In the paper

“In [52], a Gaussian Mean Field (GMF) network was proposed and combined with CNNs where all the parameters are trainable.”

What the sources say

• Not checked yetcites [52] Vemulapalli R., Tuzel O., Liu M.-y., and Chellappa R., “ Gaussian conditional random field network for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3224– 3233. source

In the paper

“In [54], a generic CNN-CRF model was proposed to handle arbitrary potentials for labeling body parts in depth images. However, it has not yet been validated with other segmentation applications.”

What the sources say

• Not checked yetcites [54] Kirillov A., Zheng S., Schlesinger D., Forkel W., Zelenin A., Torr P., and Rother C., “ Efficient likelihood learning of a generic CNN-CRF model for semantic segmentation,” arXiv preprint arXiv:1511.05067. source

In the paper

“GeoF [41] uses the geodesic distance to encode variable dependencies in the feature space and it is combined with Random Forests for semantic segmentation. However, it is not designed to deal with user interactions.”

What the sources say

• Not checked yetcites [41] Kontschieder P., Kohli P., Shotton J., and Criminisi A., “ GeoF geodesic forests for learning coupled predictors,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 65– 72. source

In the paper

“More freeform pairwise potentials for a pair of super-pixels or image patches were proposed in [27], [53], but such CRFs have a low resolution.”

What the sources say

• Not checked yetcites [27] Lin G., Shen C., Reid I., and van dan Hengel A., “ Efficient piecewise training of deep structured models for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3194– 3203. source
• Not checked yetcites [53] Liu F., Shen C., and Lin G., “ Deep convolutional neural fields for depth estimation from a single image,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 5162– 5170. source

In the paper

“However, the pair-wise potentials in [50] are limited to weighted Gaussians and not all the parameters are trainable due to the Permutohedral lattice implementation [51].”

What the sources say

• Not checked yetcites [50] Zheng S., Jayasumana S., Romera-Paredes B., Vineet V., Su Z., Du D., Huang C., and Torr P. H. S., “ Conditional random fields as recurrent neural networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529– 1537. source
• Not checked yetcites [51] Adams A., Baek J., and Davis M. A., “ Fast high-dimensional filtering using the permutohedral lattice,” Comput. Graph. Forum, vol. 29, no. 2, pp. 753– 762, 2010. source

In the paper

“The ground truth was manually delineated by an experienced Radiologist.”

In the paper

“The 2D segmentation results can also be used for motion correction and high-resolution volume reconstruction [63].”

What the sources say

• Not checked yetcites [63] Keraudren K., Kuklisova-Murgasova M., Kyriakopoulou V., Malamateniou C., Rutherford M. A., Kainz B., Hajnal J. V., and Rueckert D., “ Automated fetal brain segmentation from 2D MRI slices for motion correction,” NeuroImage, vol. 101, pp. 633– 643, 2014. 25058899 source

In the paper

“It is more memory efficient and therefore can perform inference on a 3D volume in interactive time.”

In the paper

“Differently from previous works [55] that re-train the learning model each time when new user interactions are given, the proposed R-Net is only trained with user interactions once since it takes a considerable time to re-train a CNN model with a large training set.”

What the sources say

• Not checked yetcites [55] Wang G., Zuluaga M. A., Pratt R., Aertsen M., Doel T., Klusmann M., David A. L., Deprest J., Vercauteren T., and Ourselin S., “ Dynamically balanced online random forests for interactive scribble-based segmentation,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2016, pp. 352– 360. source

In the paper

“Continuous max-flow [43] can also be used for the minimization, but its parameters are manually designed.”

What the sources say

• Not checked yetcites [43] Yuan J., Bae E., and Tai X. C., “ A study on continuous max-flow and min-cut approaches,” in Proc. IEEE Comput. Society Conf. Comput. Vis. Pattern Recognit., 2010, pp. 2217– 2224. source

In the paper

“Two users (an Obstetrician and a Radiologist) respectively used these interactive methods to segment every test image until the result was visually acceptable.”

In the paper

“In [50], a CRF based on RNN was proposed and it can be trained by back-propagation.”

What the sources say

• Not checked yetcites [50] Zheng S., Jayasumana S., Romera-Paredes B., Vineet V., Su Z., Du D., Huang C., and Torr P. H. S., “ Conditional random fields as recurrent neural networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529– 1537. source

In the paper

“The CRF-RNN network [50] formulated dense CRFs as RNNs so that the CNNs and CRFs can be jointly trained in an end-to-end system for segmentation.”

What the sources say

• Not checked yetcites [50] Zheng S., Jayasumana S., Romera-Paredes B., Vineet V., Su Z., Du D., Huang C., and Torr P. H. S., “ Conditional random fields as recurrent neural networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529– 1537. source

In the paper

“P-Net and R-Net use a resolution-preserving structure that captures high-level features from a large receptive field without loss of resolution.”

In the paper

“Second, though long-distance dependency helps to improve segmentation in most RGB images [22], [45], [50]”

What the sources say

• Not checked yetcites [22] Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “ Semantic image segmentation with deep convolutional nets and fully connected CRFs,” arXiv preprint arXiv:1412.7062. 28463186 source
• Not checked yetcites [45] Krähenbühl P. and Koltun V., “ Efficient inference in fully connected CRFs with gaussian edge potentials,” in Proc. Neural Inf. Process. Syst. Conf., 2011, pp. 109– 117. source
• Not checked yetcites [50] Zheng S., Jayasumana S., Romera-Paredes B., Vineet V., Su Z., Du D., Huang C., and Torr P. H. S., “ Conditional random fields as recurrent neural networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529– 1537. source

In the paper

“It is fast due to accessing the image memory in contiguous blocks.”

What the sources say

• Not checked yetcites [36] Criminisi A., Sharp T., and Blake A., “ GeoS: Geodesic image segmentation,” in Proc. Eur. Conf. Comput. Vis., 2008, pp. 99– 112.