Automatic breast ultrasound image segmentation: A survey

“Let w be the normal vector to the hyperplane, and { = (0, 1 − ( ∙ + ))} be slack variables for the soft margins; then the problem can be formulated as a constrained quadratic optimization problem [97] 1 + ‖ ‖ (30)”

“[115] Global thresholding ROI generation Otsu’s algorithm [130] Depended on image preprocessing.”

“Deep learning: in the last several years, deep learning has demonstrated impressive performance for many tasks such as object recognition [158], image classification [159], semantic segmentation [160], facial expression recognition [163], speech recognition [164], medical applications [161, 170], etc.”

“Graph-based approaches Graph-based approaches gain increasing popularity in BUS image segmentation because they offer several advantages: (1) they provide a simple way to organize task-related priors and image information in a unified framework; (2) they are flexible and suitable for expressing soft constraints betwee…”

“Let = ( , ℰ ) be the subgraph corresponding to non-tumor regions, and = ( , ℰ ) be the subgraph corresponding to the tumor regions, where , ⊆ and ℰ , ℰ ⊆ ℰ, ∪ = , ∩ = ∅, and ℰ ∩ ℰ = ∅.”

“The general formulation of P in the original DMs [38] is defined by ( , ) = ( , ) + ( , ) + ( , ) (18) where , and extract image line, edge and termination features, respectively; while , and are the corresponding weights.”

“In [28], both the region-level [29] and pixellevel features were utilized to define the weights in the data term; and all weights were learned by using a structural SVM [30].”

“They proposed a BUS image segmentation approach based on the active contour without edges (ACWE) model [66] which defined the stopping term using Mumford-Shah technique and was robust to segment images with weak boundaries.”

“In [28], both the region-level [29] and pixellevel features were utilized to define the weights in the data term; and all weights were learned by using a structural SVM [30].”

“For state-of-the-art superpixel generation approaches, refer [177-179].”

“, mean filter, Gaussian low-pass filter, speckle reducing anisotropic diffusion (SRAD) [148], nonlinear coherence diffusion (NCD) [149], sticks filter [60], bilateral filter [150, 151], fractional subpixel diffusion [157], etc.”

“For more details about applying deep learning to medical image analysis, refer the survey paper [171].”

“The energy function of MRF-MAP can be optimized by using Simulated Annealing (SA) [14] and Iterated Conditional Mode (ICM) [15] algorithms.”

“Learning-based approaches thrive in BUS image segmentation in the last decade and we believe new deep learning techniques [162] such as deep convolutional neural networks (CNNs) and recurrent neural network (RNN) will make great progress in segmenting BUS images in the near future.”

“Yezzi et al. [42] modified traditional GDMs to have an additional term (〈∇ , 〉) that provided stronger stopping power at object edges.”

“Second, random walk [173] was applied to generate the final tumor boundary by placing seed at the center of each true region.”

“The segmentation of an image is to find a cut that minimizes ( ); the minimum s-t cut is equivalent to maximize the flow (max-flow) from s to t according to the Ford-Fulkerson theorem [19].”

“Balloon model [39] ∗ = + is the unit vector to the curve, and k is the amplitude Increased attraction range; can find the minima even the initial contour is smaller than the minima contour; limited to extract 2D curve Cannot handle concave boundary; difficult to set the strength of the pressure; amply the issue of ign…”

“For more information about the reinitialization, please refer [45-47].”

“[148], 2013 GDMs F 861 Region-based GDM and solve the inhomogeneity problem better Slow and sensitive to initialization”

“The approach utilized user interaction to generate the volume of interest (VOI), applied the radial-gradient index [83] to estimate the initial lesion boundary, and implemented the region-based DM [48, 49] iteratively to find the final contour.”

“Yezzi et al. [42] modified traditional GDMs to have an additional term (〈∇ , 〉) that provided stronger stopping power at object edges.”

“The approach utilized user interaction to generate the volume of interest (VOI), applied the radial-gradient index [83] to estimate the initial lesion boundary, and implemented the region-based DM [48, 49] iteratively to find the final contour.”

“In [28], both the region-level [29] and pixellevel features were utilized to define the weights in the data term; and all weights were learned by using a structural SVM [30].”

“Introduction Breast cancer occurs in the highest frequency in women among all cancers, and is also one of the leading causes of cancer death worldwide [1, 2].”

“[145, 146] applied RGI to find the initial tumor contour of 3D BUS image.”

“[87] proposed a BUS image segmentation method using the spatial FCM (sFCM) [88] with local texture and intensity features.”

“Deep learning: in the last several years, deep learning has demonstrated impressive performance for many tasks such as object recognition [158], image classification [159], semantic segmentation [160], facial expression recognition [163], speech recognition [164], medical applications [161, 170], etc.”

“For state-of-the-art superpixel generation approaches, refer [177-179].”

“25 / 71 For the points in the low contrast class, both the global and local region-based energies [66, 77] are used; while for the high contrast class, only the local region-based [77] energy is utilized.”

“2 Graph cuts Graph cuts was proposed to solve a special case in MRF-MAP framework [16]: the global optimization of the binary labelling problem (L = { , }).”

“The approach was compared with two other GDM approaches [66, 68] using 79 BUS images.”

“For more information about the reinitialization, please refer [45-47].”

“The most common implementation of watershed for image segmentation can be found in [121].”

“, mean filter, Gaussian low-pass filter, speckle reducing anisotropic diffusion (SRAD) [148], nonlinear coherence diffusion (NCD) [149], sticks filter [60], bilateral filter [150, 151], fractional subpixel diffusion [157], etc.”

“Deep learning: in the last several years, deep learning has demonstrated impressive performance for many tasks such as object recognition [158], image classification [159], semantic segmentation [160], facial expression recognition [163], speech recognition [164], medical applications [161, 170], etc.”

“Breast ultrasound (BUS) imaging has become one of the most important and effective modality for the early detection of breast cancer because of its noninvasive, nonradioactive and cost-effective nature [4]; and it is most suitable for large scale breast cancer screening and diagnosis in low-resource countries and regi…”

“Gao et al. [73] proposed a level set approach for BUS image segmentation based on the method in [47] by redefining the edge-based stop function using phase congruency [74] which is invariant to intensity magnitude, and integrated GVF model into the level set framework.”

“, mean filter, Gaussian low-pass filter, speckle reducing anisotropic diffusion (SRAD) [148], nonlinear coherence diffusion (NCD) [149], sticks filter [60], bilateral filter [150, 151], fractional subpixel diffusion [157], nonlocal means-based speckle filter [167], etc.”

“For more details about modern image denoising or filtering approaches, please refer [168].”

“In edge-based DMs, [42, 50, 52, 56 - 59, 61 - 63], image gradient is applied to constructing the external force or speed function of the evolving curve; as discussed in section 3.”

“It was then improved to solve the general (color image segmentation) and multi-label problem [17, 18].”

“Edge-based DMs aim to attract the evolving curve to object boundary by defining the deformation force or speed using image gradient [42-44], and the models depend on image gradient to stop curve deformation.”

“[71] considered the SNR and local intensity value as two important features for estimating tumor boundary, and built a two-fold termination criterion based on the two features in discrete dynamic DMs [72].”

“In [142], the contour with max ARD value [25,26] was chosen as the final tumor contour.”

“[73] proposed a level set approach for BUS image segmentation based on the method in [47] by redefining the edge-based stop function using phase congruency [74] which is invariant to intensity magnitude, and integrated GVF model into the level set framework.”

“A detailed comparison between ICM and other MRF energy minimization techniques can be found in [35].”

“, in [120], a well-trained ANN was applied to perform the tumor segmentation.”

“In graph-based models, it corresponds the smoothness term (pairwise energy), and the minimization of the energy makes the models produce a smooth boundary; however, it is important to notice that the smoothness term also makes the models have the tendency to shrink and generate much shorter boundary than the real boun…”

“(41)) which was proposed in [141] to segment mammograms.”

“The first is to choose empirical value as the threshold for the whole dataset [109-111]; the second approach is to select the threshold for each image based on domain related rules [112, 113]; and the third is to generate the threshold automatically based on statistical-decision theory [22, 114, 115].”

“In [142], the contour with max ARD value [25,26] was chosen as the final tumor contour.”

“[175] is the first survey paper for BUS image segmentation published recently; however, this paper is much more comprehensive in terms of major issue discussions, future direction prediction, theory fundamentals, application theme and the number of references.”

“Cellular automata (CA): CA was introduced by von Neumann [133] and applied to interactive image segmentation [134].”

“[122] applied the watershed to segment the preprocessed BUS images and the markers were selected based on grey level and connectivity.”

“22 / 71 dual snake [53] to handle the two problems of applying traditional DMs for BUS image segmentation: (1) initial contour should be placed close to the tumor boundary; and (2) cannot capture highly winding tumor boundary.”

“In order to improve the process of manual selection of Gaussian parameters in [11], reference [12] proposed a one-click user interaction to estimate Gaussian parameters automatically.”

“[124] Watershed Pre-segmentation Decided the external and internal markers by computing the Beucher gradient [131] of the morphological dilation and erosion of the binary image Depended on image preprocessing; select 255 groups of markers by thresholding image using thresholds from 0 to 255; needed additional geometri…”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“Texture and other local region features have more descriptive power than intensity, and have been studied in many works [9, 10, 64, 98] that they can distinguish tumor regions from normal tissues with high accuracy.”

“[62] and some early works [58-61] detected tumor automatically based on low-level image features, e.”

“[67] model the probability density difference between the intensity distribution of tumor/background region and estimated Rayleigh distribution to improve the performance of GDM.”

“46 / 71 In ultrasound imaging, speckle noise is inherent to coherent illumination and Rayleigh scattering caused by tissue microstructures [147], and it is a major difficulty in BUS image segmentation.”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“Texture and other local region features have more descriptive power than intensity, and have been studied in many works [9, 10, 64, 98] that they can distinguish tumor regions from normal tissues with high accuracy.”

“NBC produced a set of suspected lesions, and a two-phase lesion selection method based on region shape features (area size, width-height ratio, region ratio and compactness) and region continuity and volume size were applied for final tumor region decision [108].”

“[94, 95] extended the FCM and proposed the neutrosophic l-means (NLM) clustering which takes”

“255 groups of markers were selected by thresholding (th = 1, 2, ⋯, 255) the image [124, 125]; the external and the internal markers were defined by using the morphological dilation and erosion.”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[110, 111] Global thresholding Pre-segmentation Fixed threshold Depended on image enhancement; could not adapt to variations of image quality.”

“[62] and some early works [58-61] detected tumor automatically based on low-level image features, e.”

“The stopping criterion was defined as ( ̅ − ̅ ) − ( ̅ − ̅ ) = 0 [84], where ̅ and ̅ were the mean intensities inside and outside the segmented regions at step t, respectively.”

“DMs are proposed by Terzopoulos [37], and then become a popular and active field after the snake approach for planar image proposed by Kass et al.”

“[9] stated that healthy and pathological breast tissues present different textures of BUS images, and improved the method in [7] by modeling both intensity and texture distributions in the likelihood energy; they also assume that the texture features represented by using co-occurrence matrix follow the Gaussian distri…”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[127] Watershed Pre-segmentation Used regions on the binary edge map as markers Depended on empirical rules to refine the results [128] Watershed Pre-segmentation Local intensity minima Depended on empirical rules to refine the results In this section, we present the theoretic background of three classical image segme…”

“[9] stated that healthy and pathological breast tissues present different textures of BUS images, and improved the method in [7] by modeling both intensity and texture distributions in the likelihood energy; they also assume that the texture features represented by using co-occurrence matrix follow the Gaussian distri…”

“, computed tomography (CT), magnetic resonance imaging (MRI), mammography, BUS image segmentation is very challenging because (1) ultrasound image has very low quality due to the speckle noise, low contrast, low single noise ratio (SNR) and artifacts; (2) large variations of breast structures exists among patients, wh…”

“[33] proposed a semi-automatic BUS image segmentation method based on Normalized cut.”

“In [56, 57], Sahiner applied PDM for 3D BUS tumor segmentation, the external forces have two terms; the first term is defined on image gradient calculated using 3×3 Sobel filters; and the second term is the balloon force.”

“[91] 2014 FCM F 148 High sensitive rate Sensitive to initialization; used fixed threshold 2.”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“Texture and other local region features have more descriptive power than intensity, and have been studied in many works [9, 10, 64, 98] that they can distinguish tumor regions from normal tissues with high accuracy.”

“, computed tomography (CT), magnetic resonance imaging (MRI), mammography, BUS image segmentation is very challenging because (1) ultrasound image has very low quality due to the speckle noise, low contrast, low single noise ratio (SNR) and artifacts; (2) large variations of breast structures exists among patients, wh…”

“[62] and some early works [58-61] detected tumor automatically based on low-level image features, e.”

“[63] proposed a fast GDM method for biomedical image segmentation.”

“3, because of the speckle noise and week boundary problems, the performance of most approaches depends on both denoising and edge preservation techniques; [73, 75] defined the stop function of GDM for edge detection results in the frequency domain rather than in the spatial domain, which makes the GDM insensitive to i…”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[62] and some early works [58-61] detected tumor automatically based on low-level image features, e.”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[100] proposed two-step BUS segmentation approach.”

“[22, 115, 154] obtain fully automatic BUS image segmentation modeling biological priors of BUS images or visual attention mechanism, which open new avenues for fully automatic BUS”

“The category of the others [136, 138, 139, 140, 142-146] is composed of three small sub-categories, each contains only few literatures.”

“, mean filter, Gaussian low-pass filter, speckle reducing anisotropic diffusion (SRAD) [148], nonlinear coherence diffusion (NCD) [149], sticks filter [60], bilateral filter [150, 151], fractional subpixel diffusion [157], etc.”

“[124] Watershed Pre-segmentation Decided the external and internal markers by computing the Beucher gradient [131] of the morphological dilation and erosion of the binary image Depended on image preprocessing; select 255 groups of markers by thresholding image using thresholds from 0 to 255; needed additional geometri…”

“For state-of-the-art superpixel generation approaches, refer [177-179].”

“[23], 2015 Graph cuts F 184 images No distribution assumption; model the tumor pose, position; construct smoothness term in both spatial and frequency domains “shrink” problem”

“The intensity and sensitivity of interaction are two important criteria for evaluating interactive segmentation approaches [152, 153]; because user interaction has large degree of arbitrariness and different interaction may lead to quite different results.”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[27, 34] applied normalized cut as an initial step to segment BUS images into small patches, and the final segmentation needs extra merging step and formulated knowledge to determine the tumor region.”

“[32], 2012 Normalized cut S 100 images No shrink problem Manually select ROI”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[81, 82] proposed a semi-automatic approach for 3D BUS image segmentation.”

“[107], 2012 NBC F 31 cases (Whole BUS images) Achieved high sensitive ratio Depended on the assumption of intensity distribution; depended on post selection to reduce false positive ratio 4.”

“The number of classes is usually set from 2 to 4 for BUS image; and β could be set adaptively [7] or set to a constant [11-13]; and the parameters of Gaussian distribution are usually initialized by using K-means algorithm [7, 9, 10].”

“[110, 111] Global thresholding Pre-segmentation Fixed threshold Depended on image enhancement; could not adapt to variations of image quality.”

“[127] Watershed Pre-segmentation Used regions on the binary edge map as markers Depended on empirical rules to refine the results [128] Watershed Pre-segmentation Local intensity minima Depended on empirical rules to refine the results In this section, we present the theoretic background of three classical image segme…”

“[27, 34] applied normalized cut as an initial step to segment BUS images into small patches, and the final segmentation needs extra merging step and formulated knowledge to determine the tumor region.”

“[81, 82] proposed a semi-automatic approach for 3D BUS image segmentation.”

“Human vision system is very sensitive to the topological properties of objects, and some works have been investigated [115, 152, 153] for both natural and medical image processing tasks.”

“[104] trained an ANN to generate the threshold for BUS image segmentation.”

“The approaches [54, 55, 113, 106] constructing empirical formulas to model domain priors provided effective means for tumor detection, but the fixed RP formulated in these approaches limited their robustness and flexibility.”

“35 / 71 [103] ANN F 93 cases (3D) Fully automatic Depended on fixed threshold to produce candidate regions; relatively low performance [106], 2012 ANN F 120 images Achieved good performance by using the phase information, radial distance and the joint distribution of texture and intensity Depended on fixed reference p…”

“In [22, 54, 55], no explicit distribution was predefined, and histogram was applied to describe the distribution of tumor region and normal tissues; In [25, 27, 28], supervised learning approaches are introduced to train the classifiers to output the probability of each image region to be tumor or background.”

“[124], 2012 PDMs S 20 Redefine the edge-based stopping function using phase information Validated only on a small dataset; sensitive to noise [139], 2013 GDMs S 168 Use both local statistics and phase information to define the speed function; and handle weak boundary and inhomogeneity problems better Slow”

“Potential directions are to learn invariant and discriminative representations of tumors in BUS images, and to model human vision mechanism [110, 154, 176].”

“[117] defined the cost of growing a region by modelling common contour smoothness and region similarity (mean intensity and size): ( , ) = ( ) − ( ) ∙ ( , ) + ∙ ( ) ∙ ( ) ( ) + ( ) , ∈ ( ) (39) where (∙) denotes the mean intensity of a region, (∙) is the pixel number of the region, ( , ) is the length of the common co…”

“[80] proposed a new level set based DM approach; the local divergence and a smoothing kernel were introduced to improve the speed function.”

“[118] used this method for seed generation.”

“No quantitative evaluation of the segmentation [59], 2007 PDMS F 360 Use GVF to extend the attraction range and to handle concave boundaries Fixed threshold and reference point are applied for initial boundary generation; no quantitative evaluation [69], 2009 PDMs F 103 Use a well-trained texture classifier to detect…”

“3, because of the speckle noise and week boundary problems, the performance of most approaches depends on both denoising and edge preservation techniques; [73, 75] defined the stop function of GDM for edge detection results in the frequency domain rather than in the spatial domain, which makes the GDM insensitive to i…”

“[140] extended the approach to segment 3D BUS image, and applied Graph cuts to image regions for finding the final tumor boundary.”

“[71] considered the SNR and local intensity value as two important features for estimating tumor boundary, and built a two-fold termination criterion based on the two features in discrete dynamic DMs [72].”

“[79] adopted the region-based approach in [78], proposed a learning based approach (multivariate linear regression and support vector regression) to produce the parameters adaptively, and proved better performance using 481 BUS images.”

“4 / 40 signed to deal with speckle noise, and because of their excellent property in edge preservation, these approaches are employed in BUS segmentation popularly [34, 58, 65, 89, 95, 103, 105, 106, 113, 114, 124, 125, 158, 165, 166].”

“Mean filter and Gaussian low-pass filter are simple and fast, and widely used in early BUS segmentation approaches [22, 52, 61, 59, 62, 63, 109, 115, 112, 123, 133, 138]; SRAD, NCD, sticks filter, and fractional subpixel diffusion are specially designed to deal with speckle noise, and because of their excellent proper…”

“[87] proposed a BUS image segmentation method using the spatial FCM (sFCM) [88] with local texture and intensity features.”

“In order to improve the process of manual selection of Gaussian parameters in [11], reference [12] proposed a one-click user interaction to estimate Gaussian parameters automatically.”

“Seed generation: the seeds can be placed by user interactively [116, 117] or generated automatically [54, 55, 113, 118].”

“[145, 146] applied RGI to find the initial tumor contour of 3D BUS image.”

“Potential directions are to learn invariant and discriminative representations of tumors in BUS images, and to model human vision mechanism [110, 154, 176].”

“[109, 123] used watershed to segment ROI into small regions, and used the predefined criteria (area, mean intensity, geodesic center, etc) to determine the final tumor region.”

“[76] modified the local region-based level set approach [77] by using additional constrained energies centered at four markers specified by radiologist.”

“In [28], both the region-level [29] and pixellevel features were utilized to define the weights in the data term; and all weights were learned by using a structural SVM [30].”

“In [143, 144], a NN with five 5 hidden units was used to classify the candidate lesions into true positive and false positive; Kuo et al.”

“In [143, 144], a NN with five 5 hidden units was used to classify the candidate lesions into true positive and false positive; Kuo et al.”

“[79] adopted the region-based approach in [78], proposed a learning based approach (multivariate linear regression and support vector regression) to produce the parameters adaptively, and proved better performance using 481 BUS images.”

“In [56, 57], Sahiner applied PDM for 3D BUS tumor segmentation, the external forces have two terms; the first term is defined on image gradient calculated using 3×3 Sobel filters; and the second term is the balloon force.”

“However, few focus on medical US analysis, aside from one or two papers that examine specific tasks such as breast US image segmentation [55].”

“The segmentation of breast ultrasound (BUS) images can help us characterize and localize breast tumors, which is one of the key steps in computer-aided diagnosis (CAD) [4].”

“However, automatic segmentation in US imaging is considered a challenging task due to relatively low US image contrast, speckle noise and large variations in breast mass sizes and shapes [3,4].”

“Similar applications of deep learning include automated identification and segmentation of tumors in breast ultrasound [106], [107], [108], localization of clinically relevant B-line artifacts in lung ultrasonography [109], and”