Learning deep similarity metric for 3D MR–TRUS image registration
Open accessAbstract
Purpose The fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images for guiding targeted prostate biopsy has significantly improved the biopsy yield of aggressive cancers. A key component of MR-TRUS fusion is image registration. However, it is very challenging to obtain a robust automatic MR-TRUS registration due to the large appearance difference between the two imaging modalities. The work presented in this paper aims to tackle this problem by addressing two challenges: (i) the definition of a suitable similarity metric and (ii) the determination of a suitable optimization strategy.Methods This work proposes the use of a deep convolutional neural network to learn a similarity metric for MR-TRUS registration. We also use a composite optimization strategy that explores the solution space in order to search for a suitable initialization for the second-order optimization of the learned metric. Further, a multi-pass approach is used in order to smooth the metric for optimization.Results The learned similarity metric outperforms the classical mutual information and also the state-of-the-art MIND feature-based methods. The results indicate that the overall registration framework has a large capture range. The proposed deep similarity metric-based approach obtained a mean TRE of 3.86 mm (with an initial TRE of 16 mm) for this challenging problem.Conclusion A similarity metric that is learned using a deep neural network can be used to assess the quality of any given image registration and can be used in conjunction with the aforementioned optimization framework to perform automatic registration that is robust to poor initialization.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Storn, R., & Price, K. (1997) · Journal of Global Optimization · doi:10.1023/a:1008202821328 · Cited by 29,569 · see citing works
Siegel, R. L., Miller, K. D., & Jemal, A. (2018) · CA A Cancer Journal for Clinicians · doi:10.3322/caac.21442 · Cited by 16,632 · see citing works
A Rapidly Convergent Descent Method for Minimization
Fletcher, R., & Powell, M. J. D. (1963) · The Computer Journal · doi:10.1093/comjnl/6.2.163 · Cited by 4,626 · see citing works
Multimodality image registration by maximization of mutual information
Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., & Suetens, P. (1997) · IEEE Transactions on Medical Imaging · doi:10.1109/42.563664 · Cited by 4,547 · see citing works
Multi-modal volume registration by maximization of mutual information
Wells, W. M., Viola, P. A., Atsumi, H., Nakajima, S., & Kikinis, R. (1996) · Medical Image Analysis · doi:10.1016/s1361-8415(01)80004-9 · Cited by 1,961 · see citing works
Learning to compare image patches via convolutional neural networks
Zagoruyko, S., & Nikos Komodakis, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299064 · Cited by 1,540 · see citing works
Siddiqui, M. M., Rais‐Bahrami, S., Türkbey, B., George, A. K., Rothwax, J. T., Shakir, N. A., Okoro, C., Raskolnikov, D., Parnes, H. L., Linehan, W. M., et al. (2015) · JAMA · doi:10.1001/jama.2014.17942 · Cited by 1,503 · see citing works
Self-citationMIND: Modality independent neighbourhood descriptor for multi-modal deformable registration
Heinrich, M. P., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F. V., Brady, M., & Schnabel, J. A. (2012) · Medical Image Analysis · doi:10.1016/j.media.2012.05.008 · Cited by 703 · see citing works
Pinto, P. A., Chung, P. H., Rastinehad, A. R., Baccala, A. A., Kruecker, J., Benjamin, C. J., Xu, S., Yan, P., Kadoury, S., Chua, C., et al. (2011) · The Journal of Urology · doi:10.1016/j.juro.2011.05.078 · Cited by 450 · see citing works
Self-citationConditioning of Quasi-Newton Methods for Function Minimization
Shanno, D. F. (1970) · Mathematics of Computation · doi:10.2307/2004840 · Cited by 394 · see citing works
A Deep Metric for Multimodal Registration
Simonovsky, M., Gutiérrez-Becker, B., Mateus, D., Navab, N., & Nikos Komodakis, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46726-9_2 · Cited by 224 · see citing works
Automatic ultrasound–MRI registration for neurosurgery using the 2D and 3D LC2 Metric
Fuerst, B., Wein, W., Müller, M., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2014.04.008 · Cited by 101 · see citing works
Multimodal image-guided prostate fusion biopsy based on automatic deformable registration
Zettinig, O., Shah, A. M., Hennersperger, C., Eiber, M. J., Kroll, C., Kübler, H. R., Maurer, T., Milletarì, F., Rackerseder, J., zu Berge, C. S., et al. (2015) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-015-1233-y · Cited by 77 · see citing works
Calio, B. P., Sidana, A., Sugano, D., et al. (2017) · Prostate Cancer and Prostatic Diseases · doi:10.1038/pcan.2017.34 · Cited by 60 · see citing works
Improved medical image fusion based on cascaded PCA and shift invariant wavelet transforms
Benjamin, J. R., & Jayasree, T. (2017) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-017-1692-4 · Cited by 53 · see citing works
Open-source image registration for MRI–TRUS fusion-guided prostate interventions
Fedorov, A., Khallaghi, S., Sánchez, C. A., et al. (2015) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-015-1180-7 · Cited by 52 · see citing works
Khallaghi, S., Sánchez, C. A., Rasoulian, A., Sun, Y., Imani, F., Khojaste, A., Göksel, O., Romagnoli, C., Abdi, H., Chang, S. D., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2440253 · Cited by 38 · see citing works
Three-Dimensional Nonrigid MR-TRUS Registration Using Dual Optimization
Sun, Y., Yuan, J., Qiu, W., Rajchl, M., Romagnoli, C., & Fenster, A. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2375207 · Cited by 35 · see citing works
Efficient Convex Optimization Approach to 3D Non-rigid MR-TRUS Registration
Sun, Y., Yuan, J., Rajchl, M., Qiu, W., Romagnoli, C., & Fenster, A. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40811-3_25 · Cited by 30 · see citing works
Sparks, R. E., Bloch, B., Feleppa, E. J., et al. (2013) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2007610 · Cited by 23 · see citing works
Validation of MRI to TRUS registration for high-dose-rate prostate brachytherapy
Poulin, E., Boudam, K., Pintér, C., Kadoury, S., Lassó, A., Gabor Fichtinger, A., & Ménard, C. M. (2018) · Brachytherapy · doi:10.1016/j.brachy.2017.11.018 · Cited by 21 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [10]Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks In: Proceedings of the 25th international conference on neural information processing systems, vol 1, Curran Associates Inc., USA, NIPS’12, pp 1097– 1105
Cited by
Sorted by citations
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058 · see citing works
Deep learning in medical image registration: a survey
Haskins, G., Kruger, U., & Yan, P. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01060-x · Cited by 704 · see citing works
Self-citationDeep learning in medical image registration: a review
Fu, Y., Lei, Y., Wang, T., Curran, W. J., Liu, T., & Yang, X. (2020) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ab843e · Cited by 661 · see citing works
Medical image registration using deep neural networks: A comprehensive review
Boveiri, H. R., Khayami, R., Javidan, R., & Mehdizadeh, A. (2020) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2020.106767 · Cited by 179 · see citing works
Fu, Y., Lei, Y., Wang, T., Patel, P., Jani, A. B., Mao, H., Curran, W. J., Liu, T., & Yang, X. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101845 · Cited by 75 · see citing works
- Authors
- Haskins, G., Kruecker, J., Kruger, U., Xu, S., Pinto, P. A., Wood, B. J., & Yan, P.
- Affiliations
- Rensselaer Polytechnic Institute (US)
- National Institutes of Health (US)
- Corresponding author
- Yan, Pingkun
- ORCID
- Yan, Pingkun
- Venue
- International Journal of Computer Assisted Radiology and Surgery
- Published
- 2018-10-31
- Volume
- 14
- Issue
- 3
- Pages
- 417-425
- ISSN
- 1861-6410 (ISSN-L)
- 1861-6429
- PMC
- PMC7641874
- OpenAlex
- W2808034017
- Cited by
- 140
- References
- 22
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 12 · 2023: 29 · 2022: 26 · 2021: 29 · 2020: 21 · 2019: 5 · 2018: 3
- FWCI
- 7.25
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- International Journal of Computer Assisted Radiology and Surgery (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Institutes of Health (1ZIDBC011242, 1ZIDCL040015)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Physics and Astronomy › Radiation › Advanced Radiotherapy Techniques
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- transrectal ultrasound, magnetic resonance imaging, similarity metric learning, deep convolutional neural networks, prostate biopsy guidance, multimodal image registration, mutual information
- MeSH terms
- Deep Learning, Biopsy, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging, Male, Prostate, Prostatic Neoplasms, Ultrasonography, Neural Networks, Computer, Phantoms, Imaging, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
