Pose-Dependent Weights and Domain Randomization for Fully Automatic X-Ray to CT Registration
Open accessAbstract
Fully automatic X-ray to CT registration requires a solid initialization to provide an initial alignment within the capture range of existing intensity-based registrations. This work addresses that need by providing a novel automatic initialization, which enables end to end registration. First, a neural network is trained once to detect a set of anatomical landmarks on simulated X-rays. A domain randomization scheme is proposed to enable the network to overcome the challenge of being trained purely on simulated data and run inference on real X-rays. Then, for each patient CT, a fully-automatic patient-specific landmark extraction scheme is used. It is based on backprojecting and clustering the previously trained network's predictions on a set of simulated X-rays. Next, the network is retrained to detect the new landmarks. Finally the combination of network and 3D landmark locations is used to compute the initialization using a perspective-n-point algorithm. During the computation of the pose, a weighting scheme is introduced to incorporate the confidence of the network in detecting the landmarks. The algorithm is evaluated on the pelvis using both real and simulated x-rays. The mean (± standard deviation) target registration error in millimetres is 4.1 ± 4.3 for simulated X-rays with a success rate of 92% and 4.2 ± 3.9 for real X-rays with a success rate of 86.8%, where a success is defined as a translation error of less than 30 mm .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Feature-based 2D/3D registration methods need to define meaningful landmark features manually and do not learn registration in an end-to-end fashion.
“However, these methods need to define meaningful landmark features manually and do not learn registration in an end-to-end fashion.”
From A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers (2024) · cites this paper as [17]
The introduction (page not supplied) says a prior landmark-based method required manual annotation of CT landmarks for each new patient. That supports the manual-landmark concern for that method, but not the claim that feature-based methods generally require manually defined features. The paper also describes its own approach as enabling end-to-end registration; it does not state that feature-based methods generally cannot learn registration end to end. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThere are popular feature-based 2D/3D registration methods, such as solving a PnP problem using corresponding anatomical landmarks, which may be complementary to intensity-based registration methods.
“We are aware that there are popular feature-based 2D/3D registration methods, such as solving a PnP problem using corresponding anatomical landmarks [13], [16], [17], which may be complementary to the intensity-based registration methods.”
From A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers (2024) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1× · see citing works
“Domain Randomization: Building upon the work of [10], this method proposes a new technique to bridge the domain gap, inspired by the domain randomization [14] works in robotics.”
DeepDRR – A Catalyst for Machine Learning in Fluoroscopy-Guided Procedures
Unberath, M., Zaech, J.-N., Lee, S. C., Bier, B., Fotouhi, J., Armand, M., & Navab, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00937-3_12 · Cited by 145 · cited 2× · see citing works
“One main family of works deals with simulating the image formation process of X-rays as realistic as possible [9], [10].”
Self-citationDilated FCN for Multi-Agent 2D/3D Medical Image Registration
Miao, S., Piat, S., Fischer, P., Tuysuzoglu, A., Mewes, P., Mansi, T., & Liao, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11576 · Cited by 113 · see citing works
Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration
Grupp, R. B., Unberath, M., Gao, C., Hegeman, R. A., Murphy, R. J., Alexander, C. P., Otake, Y., McArthur, B. A., Armand, M., & Taylor, R. H. (2020) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-020-02162-7 · Cited by 81 · cited 3× · see citing works
“The scenario in the present paper is similar to the scenario of [6].”
InfluentialSelf-citationPredicting Slice-to-Volume Transformation in Presence of Arbitrary Subject Motion
Hou, B., Alansary, A., McDonagh, S., Davidson, A., Rutherford, M., Hajnal, J. V., Rueckert, D., Glocker, B., & Kainz, B. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66185-8_34 · Cited by 68 · see citing works
Towards Fully Automatic X-Ray to CT Registration
Esteban, J., Grimm, M., Unberath, M., Zahnd, G., & Navab, N. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32226-7_70 · Cited by 38 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 12
- P. Markelj, D. Tomaževič, B. Likar, and F. Pernuš, "A review of 3D/2D registration methods for image-guided interventions," Med. Im. Anal., vol. 16, no. 3, pp. 642-661, 2012.
- M. J. Van der Bom, L. W. Bartels, M. J. Gounis, R. Homan, J. Timmer, M. A. Viergever, and J. P. W. Pluim, "Robust initialization of 2D- 3D image registration using the projection-slice theorem and phase correlation," Med. Phys., vol. 37, no. 4, pp. 1884-1892, 2010.
- S. Miao, A. Tuysuzoglu, Z. J. Wang, and R. Liao, "Real-time 6dof pose recovery from x-ray images using library-based drr and hybrid optimization," International Journal of Computer Assisted Radiology and Surgery, vol. 11, no. 6, pp. 1211-1220, 2016.
- J. Rackerseder, M. Baust, R. Göbl, N. Navab, and C. Hennersperger, "Initialize globally before acting locally: Enabling landmark-free 3D US to MRI registration," in Int. Conf. Med. Im. Comput. Comput. Assist. Interv. Springer, 2018, pp. 827-835.
- B. Bier, M. Unberath, J.-N. Zaech, J. Fotouhi, M. Armand, G. Os- good, N. Navab, and A. Maier, "X-ray-transform invariant anatomical landmark detection for pelvic trauma surgery," in Int. Conf. Med. Im. Comput. Comput. Assist. Interv. Springer, 2018, pp. 55-63.
- M. Unberath, J.-N. Zaech, C. Gao, B. Bier, F. Goldmann, S. C. Lee, J. Fotouhi, R. Taylor, M. Armand, and N. Navab, "Enabling machine learning in x-ray-based procedures via realistic simulation of image formation," International journal of computer assisted radiology and surgery, vol. 14, no. 9, pp. 1517-1528, 2019.
- Q. Dou, C. Ouyang, C. Chen, H. Chen, B. Glocker, X. Zhuang, and P.-A. Heng, "Pnp-adanet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation," IEEE Access, vol. 7, pp. 99 065-99 076, 2019.
- Y. Zhang, S. Miao, T. Mansi, and R. Liao, "Unsupervised x-ray image segmentation with task driven generative adversarial networks," Medical Image Analysis, vol. 62, p. 101664, 2020.
- S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, "Convolutional pose machines," in IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4724-4732.
- A. Newell, K. Yang, and J. Deng, "Stacked hourglass networks for human pose estimation," in European conference on computer vision. Springer, 2016, pp. 483-499.
- S. Suzuki and K. Abe, "Topological structural analysis of digitized bi- nary images by border following," Comput. Vis. Graph. Image Process., vol. 30, no. 1, pp. 32-46, 1985.
- M. J. Powell, "The bobyqa algorithm for bound constrained optimization without derivatives," Cambridge NA Report NA2009/06, University of Cambridge, Cambridge, pp. 26-46, 2009.
Cited by
Sorted by citations
A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers
Gao, C., Feng, A., Liu, X., Taylor, R. H., Armand, M., & Unberath, M. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3299588 · Cited by 36 · see citing works
“Researchers have proposed deep learning models that automatically detect 2D landmarks to solve the PnP problem [12] for pose estimation [13], [14], [15], [16], [17].”
BackgroundMethodSelf-citation
- Authors
- Grimm, M., Esteban, J., Unberath, M., & Navab, N.
- Affiliations
- Technical University of Munich (DE)
- Johns Hopkins University (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2021-09
- Volume
- 40
- Issue
- 9
- Pages
- 2221-2232
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W3098238965
- Cited by
- 34
- References
- 6
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 6 · 2023: 10 · 2022: 5 · 2021: 1
- FWCI
- 1.98
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- German Federal Ministry of Research and Education (FKZ 13GW0236B)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Health Sciences › Dentistry › Oral Surgery › Dental Radiography and Imaging
- Keywords
- anatomical landmark detection, domain randomization, perspective-n-point, intensity-based registration
- MeSH terms
- Algorithms, Humans, Radiography, Random Allocation, Tomography, X-Ray Computed, X-Rays, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
