Weakly-supervised convolutional neural networks for multimodal image registration
Open accessThe DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Abstract
One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
(2010) · Cited by 14,134
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Cited by
VoxelMorph: A Learning Framework for Deformable Medical Image Registration
Balakrishnan, G., Zhao, A., Sabuncu, M. R., Guttag, J., & Dalca, A. V. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2897538 · Cited by 2,174
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M., Thai, T. C., Moore, K., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,080
A deep learning framework for unsupervised affine and deformable image registration
de Vos, B. D., Berendsen, F. F., Viergever, M. A., Sokooti, H., Staring, M., & Išgum, I. (2019) · Medical Image Analysis · doi:10.1016/j.media.2018.11.010 · Cited by 818
Deep learning in medical image registration: a survey
Haskins, G., Kruger, U., & Yan, P. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01060-x · Cited by 704
Deep learning in medical image registration: a review
Fu, Y., Lei, Y., Wang, T., Curran, W. J., Liu, T., & Yang, X. (2020) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ab843e · Cited by 661
Transformation-Consistent Self-Ensembling Model for Semisupervised Medical Image Segmentation
Li, X., Yu, L., Chen, H., Fu, C.-W., Xing, L., & Heng, P.-A. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2995319 · Cited by 543
A review of multimodal image matching: Methods and applications
Jiang, X., Ma, J., Xiao, G., Shao, Z., & Guo, X. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.012 · Cited by 532
A new era: artificial intelligence and machine learning in prostate cancer
Goldenberg, S. L., Nir, G., & Salcudean, S. E. (2019) · Nature Reviews Urology · doi:10.1038/s41585-019-0193-3 · Cited by 511
Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces
Dalca, A. V., Balakrishnan, G., Guttag, J., & Sabuncu, M. R. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.07.006 · Cited by 467
Deep learning in ultrasound imaging
van Sloun, R. J. G., Cohen, R., & Eldar, Y. C. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2932116 · Cited by 316
LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking
Li, C., Xue, W., Jia, Y., Qu, Z., Luo, B., Tang, J., & Sun, D. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3130533 · Cited by 307
Ebner, M., Wang, G., Li, W., Aertsen, M., Patel, P. A., Aughwane, R., Melbourne, A., Doel, T., Dymarkowski, S., De Coppi, P., et al. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2019.116324 · Cited by 288
Self-citationA review of multimodal medical image fusion techniques
Huang, B., Yang, F., Yin, M., Mo, X., & Zhong, C. (2020) · Computational and Mathematical Methods in Medicine · doi:10.1155/2020/8279342 · Cited by 220
Application of artificial intelligence in surgery
Zhou, X.-Y., Guo, Y., Shen, M., & Yang, G.-Z. (2020) · Frontiers of Medicine · doi:10.1007/s11684-020-0770-0 · Cited by 181
Medical image registration using deep neural networks: A comprehensive review
Boveiri, H. R., Khayami, R., Javidan, R., & Mehdizadeh, A. (2020) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2020.106767 · Cited by 179
CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
Vercauteren, T., Unberath, M., Padoy, N., & Navab, N. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2946993 · Cited by 125
“One candidate surrogate measure can be defined by enforcing segmentations of the same structures to exhibit maximal overlap after registration [41].”
BackgroundSelf-citationDeep Learning in Medical Image Registration
Chen, X., Diaz-Pinto, A., Ravikumar, N., & Frangi, A. (2020) · Progress in Biomedical Engineering · doi:10.1088/2516-1091/abd37c · Cited by 106
A review of deep learning-based deformable medical image registration
Zou, J., Gao, B., Song, Y., & Qin, J. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.1047215 · Cited by 126
“(129) Anatomical labels Prostate MRI-US 3D CNN DSC, TRE MIA 2018”
BackgroundFu, Y., Lei, Y., Wang, T., Patel, P., Jani, A. B., Mao, H., Curran, W. J., Liu, T., & Yang, X. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101845 · Cited by 75
“Related works Many methods have been proposed for MR-TRUS image registration (Hu et al., 2012; Hu et al., 2018b; Khallaghi et al., 2015a; Khallaghi et al., 2015b; Mitra et al., 2012; Sun et al., 2015; van de Ven et al., 2015; Wang et al., 2016; Wang et al., 2018; Yang et al., 2011; Yang et al.,…”
BackgroundMethodMachine learning in prostate MRI for prostate cancer: current status and future opportunities
Li, H., Lee, C. H., Chia, D., Lin, Z., Huang, W., & Tan, C. H. (2022) · Diagnostics · doi:10.3390/diagnostics12020289 · Cited by 74
“[70,71] proposed a weakly supervised learning method with CNN using manually labelled anatomical structures.”
MethodA ready-to-use machine learning tool for symmetric multi-modality registration of brain MRI
Iglesias, J. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-33781-0 · Cited by 64
A Survey on Deep Learning for Precision Oncology
Wang, C.-W., Khalil, M.-A., & Firdi, N. (2022) · Diagnostics · doi:10.3390/diagnostics12061489 · Cited by 23
Gao, X., & Zheng, G. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102322 · Cited by 13
Metadata
- Authors
- Hu, Y., Modat, M., Gibson, E., Li, W., Ghavami, N., Bonmati, E., Wang, G., Bandula, S., Moore, C. M., Emberton, M., Ourselin, S., Noble, J. A., Barratt, D. C., & Vercauteren, T.
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2018-10
- Volume
- 49
- Pages
- 1-13
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2816936748
- Cited by
- 458
- References
- 77
- FWCI
- 21.81
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.