Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation

In the paper

“(II) An implicit statistical parametrisation of the left ventricular shape via NNs for pathology classification;”

In the paper

“The applicability of our approach is shown on multi-modal cardiac datasets and public benchmarks.”

In the paper

“Our evaluation shows: (I) A sub-pixel cardiac MR image segmentation approach that, in contrast to previous CNN approaches [44], [2], is robust against slice misalignment and coverage problems;”

What the sources say

• Not checked yetcites [44] [44] P. V. Tran. A fully convolutional neural network for cardiac segmentation in short-axis MRI. arXiv preprint arXiv:1604.00494 , 2016. - source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [2] [2] M. Avendi, A. Kheradvar, and H. Jafarkhani. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. MedIA , 30:108-119, 2016. -

In the paper

“We show that the proposed approach can be easily adapted to different analysis tasks ( e.g. image enhancement, segmentation) and improve the prediction accuracy of the state-of-the-art models.”

In the paper

“The main contribution of ACNN model over the standard FCN approaches is the improved shape delineation of the LV, as it can be seen in terms of the distance error metrics.”

In the paper

“Slice by slice analysis [2], [44] significantly under-performs compared to the proposed subpixel and ACNN-Seg segmentation methods.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [2] [2] M. Avendi, A. Kheradvar, and H. Jafarkhani. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. MedIA , 30:108-119, 2016. -
• Not checked yetcites [44] [44] P. V. Tran. A fully convolutional neural network for cardiac segmentation in short-axis MRI. arXiv preprint arXiv:1604.00494 , 2016. - source

In the paper

“In particular, the dice score metrics are observed to be lower since 2D analysis can yield poor performance in basal and apical parts of the heart as shown in Fig. 6.”

In the paper

“Instead of applying these two gradient descent updates sequentially in an iterative fashion, we perform a joint update training scheme and experimentally observed better convergence.”

In the paper

“The experimental results, given in Table II, show that neural network models outperforms previous state-of-theart approaches in this public benchmark dataset although the training data size was limited to 15 image sequences.”

In the paper

“As can be seen in the bottom row of Fig. 6, without the learnt shape priors, label map predictions are more prone to imaging artefacts.”

In the paper

“(IV) Last, we demonstrate state-of-the-art 3D-US cardiac segmentation results on the CETUS'14 Benchmark.”

In the paper

“For endocardium labels, on the other hand, the difference in dice score metric is observed to be less due to the larger size of the LV blood pool compared to the myocardium.”

In the paper

“By sampling codes from the latent space across these dimensions, we observed that the network captures the variation in wall thickness and blood pool size as shown in Fig. 9.”

In the paper

“It results in reduced Hausdorff distance with marginal statistical significance”

In the paper

“In addition to the increased image quality, the ACNN-SR model is computationally more efficient in terms of run-time in comparison to the SR-CNN model [34] by a factor of 5 .”

What the sources say

• Not checked yetcites [34] [34] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero, A. de Marvao, S. Cook, D. ORegan, and D. Rueckert. Multi-input cardiac image super-resolution using convolutional neural networks. In International Conference on MICCAI , pages 246-254. Springer, 2016. - source

In the paper

“In particular, we observed that some latent dimensions are more commonly used than others in tree node splits.”

In the paper

“However, we observed that the proposed baseline sub-pixel segmentation model (3D-Seg), which merges both SR and segmentation tasks, performs better than the concatenated models.”

In the paper

“To overcome this limitation, we propose a generic training strategy that incorporates anatomical prior knowledge into CNNs through a new regularisation model, which is trained end-to-end.”

In the paper

“We show that the proposed approach can be easily adapted to different analysis tasks ( e.g. image enhancement, segmentation) and improve the prediction accuracy of the state-of-the-art models.”

In the paper

“Some of the standard clinical acquisition protocols in 3DUS and CMR still have limitations in visualising the underlying anatomy due to imaging artefacts ( e.g. cardiac motion, low slice resolution, lack of slice coverage [35]) or operatordependent errors ( e.g. shadows, signal drop-outs).”

What the sources say

• Not checked yetcites [35] [35] S. E. Petersen, P. M. Matthews, J. M. Francis, M. D. Robson, F. Zemrak, R. Boubertakh, A. A. Young, S. Hudson, P. Weale, S. Garratt, et al. UK Biobanks cardiovascular magnetic resonance protocol. Journal of cardiovascular magnetic resonance , 18(1):8, 2016. - source

In the paper

“In the context of medical imaging and neural networks, anatomical priors have not been studied in much depth, particularly in the current state-of-the-art segmentation techniques [22], [38], [11], [36].”

What the sources say

• Not checked yetcites [22] [22] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, and B. Glocker. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. MedIA , 36:61-78, 2017. - source
• Not checked yetcites [38] [38] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on MICCAI , pages 234-241. Springer, 2015. - source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [11] [11] H. Chen, Y. Zheng, J.-H. Park, P.-A. Heng, and S. K. Zhou. Iterative multi-domain regularized deep learning for anatomical structure detection and segmentation from ultrasound images. In International Conference on MICCAI , pages 487-495. Springer, 2016. -

In the paper

“The new framework encourages models to follow the global anatomical properties of the underlying anatomy ( e.g. shape, label structure) via learnt non-linear representations of the shape.”

In the paper

“State-of-the-art methods operate as pixel-wise classifiers where the training objectives do not incorporate the structure and inter-dependencies of the output.”

• Also stated in: § I. INTRODUCTION · p. 2 · plain

  “As explained in a recent NN survey paper [28], the majority of the classification and regression models utilise a pixellevel loss function ( e.g. cross-entropy or mean square error) which does not fully take into account the underlying semantic information and dependencies in the output space (e.g. class labels).”

What the sources say

• Not checked yetcites [28] [28] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. S´ anchez. A survey on deep learning in medical image analysis. arXiv preprint arXiv:1702.05747 , 2017. - source

In the paper

“As can be seen in Fig. 1, the existing state-of-the-art convolutional neural network (CNN) approaches for segmentation [44], [11] and image enhancement [34] tasks perform poorly when the input data is not self-consistent for the analysis.”

What the sources say

• Not checked yetcites [44] [44] P. V. Tran. A fully convolutional neural network for cardiac segmentation in short-axis MRI. arXiv preprint arXiv:1604.00494 , 2016. - source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [11] [11] H. Chen, Y. Zheng, J.-H. Park, P.-A. Heng, and S. K. Zhou. Iterative multi-domain regularized deep learning for anatomical structure detection and segmentation from ultrasound images. In International Conference on MICCAI , pages 487-495. Springer, 2016. -
• Not checked yetcites [34] [34] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero, A. de Marvao, S. Cook, D. ORegan, and D. Rueckert. Multi-input cardiac image super-resolution using convolutional neural networks. In International Conference on MICCAI , pages 246-254. Springer, 2016. - source

In the paper

“Incorporation of prior knowledge about organ shape and location is key to improve performance of image analysis approaches.”

• Also stated in: § I. INTRODUCTION · p. 1 · boosted

  “Incorporating prior knowledge into image segmentation algorithms has proven useful in order to obtain more accurate and plausible results as summarised in the recent survey [32].”

What the sources say

• Not checked yetcites [32] [32] M. S. Nosrati and G. Hamarneh. Incorporating prior knowledge in medical image segmentation: a survey. arXiv:1607.01092 , 2016. - source

In the paper

“In this way, we can constrain the NN training process and guide the NN to make anatomically more meaningful predictions, in particular in cases where the input image data is not informative or consistent enough”

In the paper

“Additionally, we demonstrate how the learnt deep models of 3D shapes can be interpreted and used as biomarkers for classification of cardiac pathologies”

In the paper

“Most importantly, the proposed ACNN model allows us to perform HR analysis via sub-pixel feature maps generated from low resolution (LR) input data even in the presence of motion artefacts.”

In the paper

“For this reason, incorporation of prior knowledge into cardiac image analysis could provide more accurate and reliable assessment of the anatomy, which is shown in the third column of the same figure.”

In the paper

“The cross-entropy loss function operates on individual pixel level class predictions, which does not guarantee global consistency and plausible anatomical shapes even though the segmentation network has a receptive field larger than the size of structures to be segmented.”

In the paper

“Since a perfect mapping between the intensity and label spaces is practically not achievable, the T-L learnt codes are expected to be more representative due to the inclusion of additional information.”

In the paper

“This observation can be explained by the fact that the proposed T-L network is trained with small Gaussian input noise as in the case of denoising autoencoders.”

In the paper

“To achieve this goal, we extend the standard AE model to the T-L model which enables us to obtain shape representation codes directly from the intensity space.”

In the paper

“In this equation, the second term L h e ensures that the generated segmentations are in a similar low dimensional space ( e.g. shape manifold) as the ground-truth labels.”

In the paper

“A smooth distribution of the codes ensures better supervision for the main NN model (SR, Seg) since the global gradients are back-propagated by computing the Euclidean distance between the obtained distributions.”

In the paper

“Indeed, it is the main reason why we observe such a large difference in terms of Hausdorff distance.”

In the paper

“The proposed framework relies on autoencoder and T-L network models to obtain a non-linear compact representation of the underlying anatomy, which are used as priors in segmentation.”

In the paper

“Joint training of the AE and predictor enables us to learn representations that could be extracted from both intensity and label space.”

In the paper

“The results also show that introduction of shape priors in segmentation models can be useful to tackle false-positive detections and motionartefacts.”

In the paper

“The experimental results show that the state-of-the-art NN models can benefit from the learnt priors in cases where the images are corrupted and contain artefacts.”

In the paper

“Nevertheless, the proposed ACNN-Seg does not require an additional post-processing step.”

In the paper

“With this new training objective, at testing time NNs make predictions that are in agreement with the learnt shape models of the underlying anatomy, which are referred as image priors.”

In the paper

“In that regard, our model differentiates from the VGG-Net [42] feature based training objectives [26], [21].”

What the sources say

• Not checked yetcites [42] [42] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. -
• Not checked yetcites [26] [26] C. Ledig, L. Theis, F. Husz´ ar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802 , 2016. -
• Not checked yetcites [21] [21] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , pages 694-711, 2016. -

In the paper

“VGG features tend to be more general purpose representations that are learnt from ImageNet dataset containing natural images of a large variety of objects.”

What the sources say

• Not checked yetcites [42] [42] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. -

In the paper

“This is due to the fact that ACNN-SR performs feature extraction in the low dimensional image space.”

In the paper

“the segmentations generated with the former model are strongly regularised due to the second stage AE.”

In the paper

“but the model overlooks fine details of the myocardium surface since the segmentations are generated only from the coarse level feature-maps.”

In the paper

“Additionally, in the experiments we demonstrated that the learnt codes can be used as biomarkers for classification of cardiac related pathologies and we analysed the distribution of the learnt latent space.”

In the paper

“residual connections are not used in our models ( 10 -18 layers) since they do not provide significant accuracy gains for smaller networks as reported in [26].”

What the sources say

• Not checked yetcites [26] [26] C. Ledig, L. Theis, F. Husz´ ar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802 , 2016. -

In the paper

“A generative AE model is not essential for the regularisation of the proposed segmentation and SR models.”

In the paper

“In other words, shape constraints can be learnt and utilised in standard segmentation models, as shown in ACNNSeg, without a need for additional model parameters and computational complexity.”

In the paper

“This result shows that although the AE and T-L models are not trained with the classification objective, they can still capture anatomical shape variations that are linked to cardiac related pathologies.”