CoChat
Sign Up
Cited by 30Medical Image Segmentation TechniquesDomain Adaptation and Few-Shot LearningAdvanced Image Fusion Techniques

Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion

Open access
Jaganathan, Srikrishna, Kukla, Maximilian, Wang, Jian, Shetty, Karthik, Maier, Andreas·2023·2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
DOI 10.1109/wacv56688.2023.00281
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep Learning-based 2D/3D registration enables fast, robust, and accurate X-ray to CT image fusion when large annotated paired datasets are available for training. However, the need for paired CT volume and X-ray images with ground truth registration limits the applicability in interventional scenarios. An alternative is to use simulated X-ray projections from CT volumes, thus removing the need for paired annotated datasets. Deep Neural Networks trained exclusively on simulated X-ray projections can perform significantly worse on real X-ray images due to the domain gap. We propose a self-supervised 2D/3D registration framework combining simulated training with unsupervised feature and pixel space domain adaptation to overcome the domain gap and eliminate the need for paired annotated datasets. Our framework achieves a registration accuracy of 1.83 ± 1.16 mm with a high success ratio of 90.1% on real X-ray images showing a 23.9% increase in success ratio compared to reference annotation-free algorithms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/wacv56688.2023.00281

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Augmenting training data with synthetically generated X-rays helps overcome small clinical sample sizes when training landmark extractors and pose regressors.

    “Augmenting training data with synthetically generated X-rays helps overcome the small clinical sample sizes when training landmark extractors [17, 28, 46] and pose regressors [35, 56].”

    From Intraoperative 2D/3D Image Registration via Differentiable X-Ray Rendering (2024) · cites this paper as [28]

    Section 3.2 describes training a 2D/3D registration network with simulated X-ray projections, and the Introduction says that collecting large paired, annotated datasets is prohibitive. This supports the general point that synthetic images can ease a training-data shortage, but the paper studies a registration network rather than training a landmark extractor or a pose regressor as such. It also says domain adaptation is needed because simulated and real X-rays differ. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Adrien Bardes, Jean Ponce, and Yann LeCun. Vi- creg: Variance-invariance-covariance regularization for self- supervised learning. arXiv preprint arXiv:2105.04906, 2021. · arXiv
  2. Bastian Bier, Mathias Unberath, Jan-Nico Zaech, Javad Fo- touhi, Mehran Armand, Greg Osgood, Nassir Navab, and Andreas Maier. X-ray-transform invariant anatomical land- mark detection for pelvic trauma surgery. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 55-63. Springer, 2018.
  3. Wolfgang Birkfellner, Rudolf Seemann, Michael Figl, Jo- hann Hummel, Christopher Ede, Peter Homolka, Xinhui Yang, Peter Niederer, and Helmar Bergmann. Fast drr gener- ation for 2d/3d registration. In International Conference on Medical Image Computing and Computer-Assisted Interven- tion, pages 960-967. Springer, 2005.
  4. Qi Dou, Cheng Ouyang, Cheng Chen, Hao Chen, Ben Glocker, Xiahai Zhuang, and Pheng-Ann Heng. Pnp-adanet: Plug-and-play adversarial domain adaptation network with a benchmark at cross-modality cardiac segmentation. arXiv preprint arXiv:1812.07907, 2018. · arXiv
  5. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas- cal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial train- ing of neural networks. The journal of machine learning research, 17(1):2096-2030, 2016.
  6. Cong Gao, Xingtong Liu, Wenhao Gu, Benjamin Killeen, Mehran Armand, Russell Taylor, and Mathias Unberath. Generalizing spatial transformers to projective geometry with applications to 2d/3d registration. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 329-339. Springer, 2020.
  7. Ren Hui Gong and Purang Abolmaesumi. 2d/3d registration with the cma-es method. In Medical Imaging 2008: Visu- alization, Image-Guided Procedures, and Modeling, volume 6918, pages 556-564. SPIE, 2008.
  8. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  9. Matthias Grimm, Javier Esteban, Mathias Unberath, and Nassir Navab. Pose-Dependent Weights and Domain Ran- domization for Fully Automatic X-Ray to CT Registration. IEEE Transactions on Medical Imaging, 40(9):2221-2232, 2021.
  10. Martin Groher. 2D-3D registration of vascular images. PhD thesis, Technische Universität München, 2008.
  11. Wenhao Gu, Cong Gao, Robert Grupp, Javad Fotouhi, and Mathias Unberath. Extended capture range of rigid 2d/3d registration by estimating riemannian pose gradients. In International Workshop on Machine Learning in Medical Imaging, pages 281-291. Springer, 2020.
  12. Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989- 1998. Pmlr, 2018.
  13. Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V Dalca. Synthmorph: learning contrast-invariant registration without acquired images. IEEE transactions on medical imaging, 41(3):543-558, 2021.
  14. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
  15. Srikrishna Jaganathan, Jian Wang, Anja Borsdorf, and An- dreas Maier. Learning the update operator for 2d/3d image registration. In Bildverarbeitung für die Medizin 2021, pages 117-122. Springer, 2021.
  16. Srikrishna Jaganathan, Jian Wang, Anja Borsdorf, Karthik Shetty, and Andreas Maier. Deep iterative 2d/3d registra- tion. In International Conference on Medical Image Com- puting and Computer-Assisted Intervention, pages 383-392. Springer, 2021.
  17. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
  18. Mert Asim Karaoglu, Nikolas Brasch, Marijn Stollenga, Wolfgang Wein, Nassir Navab, Federico Tombari, and Alexander Ladikos. Adversarial domain feature adaptation for bronchoscopic depth estimation. In International Confer- ence on Medical Image Computing and Computer-Assisted Intervention, pages 300-310. Springer, 2021.
  19. Everine B Kraats, Graeme P Penney, Dejan Tomaževič, Theo van Walsum, and Wiro J Niessen. Standardized eval- uation of 2d-3d registration. In International Conference on Medical Image Computing and Computer-Assisted Interven- tion, pages 574-581. Springer, 2004.
  20. Annkristin Lange and Stefan Heldmann. Multilevel 2d- 3d intensity-based image registration. In International Workshop on Biomedical Image Registration, pages 57-66. Springer, 2020.
  21. Haofu Liao, Wei-An Lin, Jiarui Zhang, Jingdan Zhang, Jiebo Luo, and S Kevin Zhou. Multiview 2d/3d rigid registration via a point-of-interest network for tracking and triangulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12638-12647, 2019.
  22. Rui Liao, Li Zhang, Ying Sun, Shun Miao, and Christophe Chefd'Hotel. A review of recent advances in registration techniques applied to minimally invasive therapy. IEEE transactions on multimedia, 15(5):983-1000, 2013.
  23. Andreas K Maier, Christopher Syben, Bernhard Stimpel, To- bias Würfl, Mathis Hoffmann, Frank Schebesch, Weilin Fu, Leonid Mill, Lasse Kling, and Silke Christiansen. Learning with known operators reduces maximum error bounds. Na- ture machine intelligence, 1(8):373-380, 2019.
  24. P. Markelj, D. Tomaževič, B. Likar, and F. Pernuš. A review of 3D/2D registration methods for image-guided interven- tions. Medical Image Analysis, 16(3):642-661, 2012.
  25. Stefan Matl, Richard Brosig, Maximilian Baust, Nassir Navab, and Stefanie Demirci. Vascular image registration techniques: A living review. Medical Image Analysis, 35:1- 17, 2017.
  26. Uroš Mitrović, Primož Markelj, Boštjan Likar, Zoran Miloševič, and Franjo Pernuš. Gradient-based 3d-2d regis- tration of cerebral angiograms. In Medical Imaging 2011: Image Processing, volume 7962, pages 533-540. SPIE, 2011.
  27. Y Otake, S Schafer, JW Stayman, W Zbijewski, G Kleinszig, R Graumann, AJ Khanna, and JH Siewerdsen. Automatic localization of vertebral levels in x-ray fluoroscopy using 3d- 2d registration: a tool to reduce wrong-site surgery. Physics in Medicine & Biology, 57(17):5485, 2012.
  28. Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European conference on computer vision, pages 319-345. Springer, 2020.
  29. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
  30. Roman Schaffert, Jian Wang, Peter Fischer, Anja Bors- dorf, and Andreas Maier. Learning an Attention Model for Robust 2-D/3-D Registration Using Point-To-Plane Cor- respondences. IEEE transactions on medical imaging, 39(10):3159-3174, 2020.
  31. Roman Schaffert, Jian Wang, Peter Fischer, Andreas Maier, and Anja Borsdorf. Robust multi-view 2-d/3-d registration using point-to-plane correspondence model. IEEE transac- tions on medical imaging, 39(1):161-174, 2019.
  32. Roman Schaffert, Markus Weiß, Jian Wang, Anja Borsdorf, and Andreas Maier. Learning-based correspondence esti- mation for 2-d/3-d registration. In Bildverarbeitung für die Medizin 2020, pages 222-228. Springer, 2020.
  33. Žiga Špiclin, Boštjan Likar, and Franjo Pernuš. Fast and robust 3d to 2d image registration by backprojection of gra- dient covariances. In International Workshop on Biomedical Image Registration, pages 124-133. Springer, 2014.
  34. Richin Sukesh, Andreas Fieselmann, Srikrishna Jaganathan, Karthik Shetty, Rainer Kärgel, Florian Kordon, Steffen Kap- pler, and Andreas Maier. Training deep learning models for 2d spine x-rays using synthetic images and annotations cre- ated from 3d ct volumes. In Bildverarbeitung für die Medizin 2022, pages 63-68. Springer, 2022.
  35. Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on com- puter vision, pages 402-419. Springer, 2020.
  36. Dejan Tomazevic, Bostjan Likar, Tomaz Slivnik, and Franjo Pernus. 3-d/2-d registration of ct and mr to x-ray images. IEEE transactions on medical imaging, 22(11):1407-1416, 2003.
  37. Moritz Venator, Fengyi Shen, Selcuk Aklanoglu, Erich Bruns, Klaus Diepold, and Andreas Maier. Dual-mode train- ing with style control and quality enhancement for road im- age domain adaptation. Proceedings -2020 IEEE Win- ter Conference on Applications of Computer Vision, WACV 2020, pages 1746-1755, 2020.
  38. Sulaiman Vesal, Mingxuan Gu, Ronak Kosti, Andreas Maier, and Nishant Ravikumar. Adapt everywhere: unsupervised adaptation of point-clouds and entropy minimization for multi-modal cardiac image segmentation. IEEE Transac- tions on Medical Imaging, 40(7):1838-1851, 2021.
  39. Jian Wang, Anja Borsdorf, Benno Heigl, Thomas Köhler, and Joachim Hornegger. Gradient-based differential ap- proach for 3-d motion compensation in interventional 2-d/3- d image fusion. In 2014 2nd International Conference on 3D Vision, volume 1, pages 293-300. IEEE, 2014.
  40. Jian Wang, Roman Schaffert, Anja Borsdorf, Benno Heigl, Xiaolin Huang, Joachim Hornegger, and Andreas Maier. Dy- namic 2-D/3-D rigid registration framework using point-to- plane correspondence model. IEEE Transactions on Medical Imaging, 36(9):1939-1954, 9 2017.
  41. Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3681-3691, 2021.
  42. Xingde Ying, Heng Guo, Kai Ma, Jian Wu, Zhengxin Weng, and Yefeng Zheng. X2ct-gan: reconstructing ct from bipla- nar x-rays with generative adversarial networks. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10619-10628, 2019.
  43. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Ma- chine Learning, pages 12310-12320. PMLR, 2021.
  44. Yue Zhang, Shun Miao, Tommaso Mansi, and Rui Liao. Task driven generative modeling for unsupervised domain adap- tation: Application to x-ray image segmentation. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 599-607. Springer, 2018.
  45. Jiannan Zheng, Shun Miao, and Rui Liao. Learning cnns with pairwise domain adaption for real-time 6dof ultrasound transducer detection and tracking from x-ray images. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 646-654. Springer, 2017.
  46. Jiannan Zheng, Shun Miao, Z Jane Wang, and Rui Liao. Pair- wise domain adaptation module for cnn-based 2-d/3-d regis- tration. Journal of Medical Imaging, 5(2):021204, 2018.
  47. Shiqiang Zheng, Xin Yang, Yifan Wang, Mingyue Ding, and Wenguang Hou. Unsupervised cross-modality domain adap- tation network for cnn-based x-ray to ct. IEEE Journal of Biomedical and Health Informatics, 2021.
  48. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223- 2232, 2017.

Cited by

Sorted by citations

Authors
Jaganathan, S., Kukla, M., Wang, J., Shetty, K., & Maier, A.
Affiliations
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • Siemens Healthineers (Germany) (DE)
ORCID
Venue
2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Publisher
IEEE
Published
2023-01
Pages
2787-2797
ISSN
2472-6737 (ISSN-L)
DOI
10.1109/wacv56688.2023.00281
OpenAlex
W4319299682
Cited by
30
References
10
Citations by year
2026: 8 · 2025: 7 · 2024: 12 · 2023: 3
FWCI
3.21
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
Keywords
2D-3D registration, self-supervised learning, domain gap, domain adaptation, interventional radiology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review