A CNN Regression Approach for Real-Time 2D/3D Registration
The DOI matches its Crossref record; 31 of the 37 references it cites were found in a registry or an index.
Abstract
In this paper, we present a Convolutional Neural Network (CNN) regression approach to address the two major limitations of existing intensity-based 2-D/3-D registration technology: 1) slow computation and 2) small capture range. Different from optimization-based methods, which iteratively optimize the transformation parameters over a scalar-valued metric function representing the quality of the registration, the proposed method exploits the information embedded in the appearances of the digitally reconstructed radiograph and X-ray images, and employs CNN regressors to directly estimate the transformation parameters. An automatic feature extraction step is introduced to calculate 3-D pose-indexed features that are sensitive to the variables to be regressed while robust to other factors. The CNN regressors are then trained for local zones and applied in a hierarchical manner to break down the complex regression task into multiple simpler sub-tasks that can be learned separately. Weight sharing is furthermore employed in the CNN regression model to reduce the memory footprint. The proposed approach has been quantitatively evaluated on 3 potential clinical applications, demonstrating its significant advantage in providing highly accurate real-time 2-D/3-D registration with a significantly enlarged capture range when compared to intensity-based methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Miao et al. [324] used synthetic X-ray images to train a five-layer CNN to register 3D models of a trans-esophageal probe, a hand implant, and a knee implant onto 2D X-ray images for pose estimation.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [324]
In the citing paper
“They determined that their model achieved an execution time of 0.1 s, representing an important enhancement against the conventional registration techniques based on intensity; moreover, it achieved effective registrations 79–99% of the time.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [324]
In the citing paper
“However, since existing large-scale hip datasets have focused on 3D image modalities and pre and postoperative radiography, rather than intraoperative fluoroscopy [12], applications of CNNs to fluoroscopy have been mostly limited to recognizing surgical instruments and tools [13-16].”
From Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration (2020) · cites this paper as [13]
In the citing paper
“Even when supervised training data is augmented with synthetic X-rays, landmark-based localization and pose regression fail to achieve consistent sub-millimeter accuracy [17, 35, 56].”
From Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering (2024) · cites this paper as [35]
In the citing paper
“Typically, a finite number of synthetic X-rays is generated for pretraining [35, 56], failing to address the central limitation of CNN-based pose regression”
From Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering (2024) · cites this paper as [35]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“The weights are initialized using the Xavier method [25], and mini-batch SGD is performed for 12,500 iterations (32 epochs).”
MethodInfluential(2010) · Cited by 14,134
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 3,752 · cited 2×
“For PEHL, the neural network was implemented with cuDNN acceleration using an open-source deep learning framework, Caffe [31].”
MethodFootprint evaluation for volume rendering
Westover, L. (1990) · Proceedings of the 17th annual conference on Computer graphics and interactive techniques · doi:10.1145/97879.97919 · Cited by 787 · cited 1×
“Another strategy is splatting, which is a voxel-based volume rendering technique that directly projects single voxels to the imaging plane [11], [12].”
BackgroundA Review of 3d/2d Registration Methods for Image-Guided Interventions
Markelj, P., Tomaževič, D., Likar, B., & Pernuš, F. (2012) · Medical Image Analysis · doi:10.1016/j.media.2010.03.005 · Cited by 746 · cited 1×
“In existing methods, accurate 2-D/3-D registration is typically achieved by intensity-based 2-D/3-D registration methods [2]–[5].”
MethodDollar, P., Welinder, P., & Perona, P. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540094 · Cited by 528 · cited 1×
“[19] proposed to train cascaded regressors on a pose-indexed feature that is only affected by the difference between the ground truth and initial pose parameters for pose estimation.”
BackgroundData fusion through cross-modality metric learning using similarity-sensitive hashing
Bronstein, M. M., Bronstein, A. M., Michel, F., & Paragios, N. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539928 · Cited by 455 · cited 1×
“Several metric learning methods have been proposed to learn similarity measures using supervised learning [14], [15].”
BackgroundFootprint evaluation for volume rendering
Westover, L. (1990) · ACM SIGGRAPH Computer Graphics · doi:10.1145/97880.97919 · Cited by 404
Learning descriptors for object recognition and 3D pose estimation
Wohlhart, P., & Lepetit, V. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298930 · Cited by 358 · cited 2×
“[18] proposed to train a Convolutional Neural Networks (CNN) to learn a pose differentiating descriptor from range images, and use -Nearest Neighbor for pose estimation.”
BackgroundMethodAcceleration techniques for GPU-based volume rendering
Kruger, J., & Westermann, R. (2004) · IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control · doi:10.1109/visual.2003.1250384 · Cited by 312
Standardized evaluation methodology for 2-D-3-D registration
van de Kraats, E. B., Penney, G. P., Tomazevic, D., van Walsum, T., & Niessen, W. J. (2005) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2005.853240 · Cited by 211 · cited 1×
“Registration Error in the projection direction (mTREproj) [29],”
Background3-D/2-D registration of CT and MR to X-ray images
Tomazevic, D., Likar, B., Slivnik, T., & Pernus, F. (2003) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2003.819277 · Cited by 167 · cited 2×
“The formation model of gradients in X-ray images has been shown in [23] as:”
MethodZollei, L., Grimson, E., Norbash, A., & Wells, W. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.991032 · Cited by 120 · cited 6×
“Because of the approximation, SHMI and DSC theoretically achieve the same or degraded accuracy compared to using original MI and CC.”
BackgroundMethodInfluentialGao, G., Penney, G., Ma, Y., Gogin, N., Cathier, P., Arujuna, A., Morton, G., Caulfield, D., Gill, J. S., Rinaldi, C. A., et al. (2011) · Medical Image Analysis · doi:10.1016/j.media.2011.05.003 · Cited by 102
Monitoring tumor motion by real time 2D/3D registration during radiotherapy
Gendrin, C., Furtado, H., Weber, C., Bloch, C., Figl, M., Pawiro, S. A., Bergmann, H., Stock, M., Fichtinger, G., Georg, D., et al. (2012) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2011.07.031 · Cited by 87 · cited 2×
“We evaluated three popular similarity measures, MI, Cross Correlation (CC) and GC, which have also been reported to be effective in recent literature [3], [4], [12].”
MethodA Review of Recent Advances in Registration Techniques Applied to Minimally Invasive Therapy
Liao, R., Zhang, L., Sun, Y., Miao, S., & Chefd'Hotel, C. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2244869 · Cited by 71 · cited 1×
“T WO-DIMENSIONAL to three-dimensional registration represents one of the key enabling technologies in medical imaging and image-guided interventions [1].”
BackgroundSelf-citationA coarse-to-fine model for 3D pose estimation and sub-category recognition
Mottaghi, R., Xiang, Y., & Savarese, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298639 · Cited by 68 · cited 1×
“In recent years, promising results on object matching for computer vision tasks have been reported using machine learning methods [18]–[21].”
Background2D/3D image registration using regression learning
Chou, C.-R., Frederick, B., Mageras, G., Chang, S., & Pizer, S. (2013) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2013.02.009 · Cited by 64 · cited 2×
“[17] computed the residual between the DRR and X-ray images as a feature and trained linear regressors to estimate the transformation parameters to reduce the residual.”
BackgroundMethodFully automated 2D–3D registration and verification
Varnavas, A., Carrell, T., & Penney, G. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.08.005 · Cited by 50
A dynamic programming approach for fast and robust object pose recognition from range images
Zach, C., Penate-Sanchez, A., & Pham, M.-T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298615 · Cited by 45 · cited 1×
“In recent years, promising results on object matching for computer vision tasks have been reported using machine learning methods [17][18][19][20].”
BackgroundBirkfellner, W., Stock, M., Figl, M., Gendrin, C., Hummel, J., Dong, S., Kettenbach, J., Georg, D., & Bergmann, H. (2009) · Medical Physics · doi:10.1118/1.3157111 · Cited by 44 · cited 3×
“However, only a few similarity measures are suitable to be calculated on a random subset of the image, e.g., Mutual Information (MI) [9] and Stochastic Rank Correlation (SRC) [10].”
BackgroundMethodMcLaughlin, R. A., Hipwell, J. H., Hawkes, D. J., Noble, J. A., Byrne, J. V., & Cox, T. C. S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45787-9_65 · Cited by 41 · cited 1×
“Intensity-based methods are known to be able to achieve high registration accuracy [6], but at the same time, they suffer from two major drawbacks: 1) long computation time and 2) small capture range.”
BackgroundBoosted metric learning for 3D multi-modal deformable registration
Michel, F., Bronstein, M. M., Bronstein, A. M., & Nikos Paragios, A. (2011) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2011.5872619 · Cited by 27 · cited 1×
“Several metric learning methods have been proposed to learn similarity measures using supervised learning [14], [15].”
BackgroundRegistration-by-regression of coronary CTA and X-ray angiography
Gouveia, A. R., Metz, C., Freire, L., Almeida, P. M. D., & Klein, S. (2015) · Computer Methods in Biomechanics and Biomedical Engineering Imaging & Visualization · doi:10.1080/21681163.2015.1054520 · Cited by 26 · cited 4×
“In [16], the linear regressors were reported to be trained on X-ray images with fixed ground truth transformation parameters, and therefore can only be applied on X-ray images with poses within a limited range.”
BackgroundMethodInfluentialMiao, S., Liao, R., & Zheng, Y. (2011) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2011.5872620 · Cited by 24 · cited 1×
“In existing methods, accurate 2-D/3-D registration is typically achieved by intensity-based 2-D/3-D registration methods [2]–[5].”
MethodSelf-citationSegmentation of X-ray Images by 3D-2D Registration Based on Multibody Physics
Schmid, J., & Chênes, C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16808-1_45 · Cited by 22 · cited 2×
“GC has been adopted in [4] for the assessing the positioning and migration of bone implants.”
BackgroundMethodRobust 5DOF Transesophageal Echo Probe Tracking at Fluoroscopic Frame Rates
Hatt, C. R., Speidel, M. A., & Raval, A. N. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_36 · Cited by 13 · cited 9×
“Because of the approximation, SHMI and DSC theoretically achieve the same or degraded accuracy compared to using original MI and CC.”
BackgroundMethodInfluential2D/3D Registration of TEE Probe from Two Non-orthogonal C-Arm Directions
Kaiser, M., John, M., Heimann, T., Brost, A., Neumuth, T., & Rose, G. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10404-1_36 · Cited by 12 · cited 1×
“parameterize the transformation by 3 in-plane and 3 out-ofplane transformation parameters [22], as shown in Fig.”
MethodToward Accurate and Robust 2-D/3-D Registration of Implant Models to Single-Plane Fluoroscopy
Miao, S., Liao, R., Lucas, J., & Chefd’hotel, C. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40843-4_11 · Cited by 10 · cited 2×
“[8] proposed to use shape encoding combined with template matching for initial pose estimation.”
MethodSelf-citationComparison of Optimizers for 2D/3D Registration for Fusion of Ultrasound and X-Ray
Kaiser, M., John, M., Heimann, T., Neumuth, T., & Rose, G. (2014) · Informatik aktuell · doi:10.1007/978-3-642-54111-7_58 · Cited by 5 · cited 1×
“A recent study compared four popular optimizers (Powell's method, Nelder-Mead, BFGS, CMA-ES) for intensity-based 2-D/3-D registration, and concluded that Powell's method achieved the best performance [30].”
MethodZhu, Z., Ji, S., Yang, M., Ding, H., & Wang, G. (2013) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-642-29305-4_62 · Cited by 3 · cited 1×
“We evaluated PEHL on datasets from the following 3 clinical applications to demonstrate its wide applicability for real-time 2-D/3-D registration: 1) Total Knee Arthroplasty (TKA) Kinematics: In the study of the kinematics of TKA, 3-D kinematics of knee prosthesis can be estimated by matching the 3-D model of the knee…”
Method
Research works printed but not linked: 1
- vetter, 2014, Validation of a virtual implant planning system (VIPS) in distal radius fractures, Bone Joint J Orthopaedic Proc Supplement, 96, 50
Cited by
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Greenspan, H., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,700
Deep Learning Applications in Medical Image Analysis
Ker, J., Wang, L., Rao, J., & Lim, T. (2018) · IEEE Access · doi:10.1109/access.2017.2788044 · Cited by 1,481
“[84] trained a 5 layer CNN on synthetic X-ray images in order to register 3-dimensional models of a knee implant, a hand implant, and a trans-esophageal probe onto 2-dimensional X-ray images, in order to estimate their pose.”
MethodZhou, S. K., Greenspan, H., Davatzikos, C., Duncan, J. S., Van Ginneken, B., Madabhushi, A., Prince, J. L., Rueckert, D., & Summers, R. M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3054390 · Cited by 1,118
“Still, some unique and varied approaches have achieved state-of-the-art results with relatively fast run times [120], [121], [122], [123].”
Background
Metadata
- Authors
- Miao, S., Wang, Z. J., & Liao, R.
- Affiliations
- University of British Columbia (CA)
- Siemens (United States) (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1352-1363
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2344328023
- Cited by
- 514
- References
- 37
- Citations by year
- 2026: 23 · 2025: 40 · 2024: 56 · 2023: 57 · 2022: 55 · 2021: 64 · 2020: 60 · 2019: 76 · 2018: 41 · 2017: 38 · 2016: 4
- FWCI
- 35.73
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Physics and Astronomy › Radiation › Advanced Radiotherapy Techniques
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Keywords
- 2D-3D registration, convolutional neural network regression, intensity-based registration, real-time registration, digitally reconstructed radiographs, X-ray images, automatic feature extraction, hierarchical regression, parameter sharing, capture range, transformation parameter estimation
- MeSH terms
- Machine Learning, Humans, Knee Joint, Radiography, Regression Analysis, Neural Networks, Computer, Echocardiography, Transesophageal, Arthroplasty, Replacement, Knee, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.