CoChat
Sign Up
Cited by 463D Shape Modeling and AnalysisMedical Image Segmentation TechniquesAdvanced Vision and Imaging

Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration

Open access
Gao, Cong, Liu, Xingtong, Gu, Wenhao, Killeen, Benjamin, Armand, Mehran, Taylor, Russell, Unberath, Mathias·2020·Lecture Notes in Computer Science
DOI 10.1007/978-3-030-59716-0_32PMC PMC7602675
Open this paper in CoChat — free account

Abstract

Differentiable rendering is a technique to connect 3D scenes with corresponding 2D images. Since it is differentiable, processes during image formation can be learned. Previous approaches to differentiable rendering focus on mesh-based representations of 3D scenes, which is inappropriate for medical applications where volumetric, voxelized models are used to represent anatomy. We propose a novel Projective Spatial Transformer module that generalizes spatial transformers to projective geometry, thus enabling differentiable volume rendering. We demonstrate the usefulness of this architecture on the example of 2D/3D registration between radiographs and CT scans. Specifically, we show that our transformer enables end-to-end learning of an image processing and projection model that approximates an image similarity function that is convex with respect to the pose parameters, and can thus be optimized effectively using conventional gradient descent. To the best of our knowledge, we are the first to describe the spatial transformers in the context of projective transmission imaging, including rendering and pose estimation. We hope that our developments will benefit related 3D research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-030-59716-0_32

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The multi-scale patch-based feature extraction design learns more robust features than the ResNet block encoders used in the MICCAI published version.

    “The multi-scale patch-based feature extraction design learns more robust features than the ResNet block encoders used in the MICCAI published version [31].”

    From A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers (2024) · cites this paper as [31]

    The Methodology section (page unavailable) describes two encoders that produce embedded features, supporting the citation’s identification of an earlier encoder-based design. Our copy does not identify them as ResNet-block encoders or compare them with multi-scale patch-based extraction; the claimed improvement in feature robustness is therefore not established by this source. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 18

  1. [1]Ferrante E, Oktay O, Glocker B, Milone DH: On the adaptability of unsupervised cnn-based deformable image registration to unseen image domains. In: International Workshop on Machine Learning in Medical Imaging pp. 294– 302. Springer (2018)
  2. [2]Gibson E, Giganti F, Hu Y, Bonmati E, Bandula S, Gurusamy K, Davidson B, Pereira SP, Clarkson MJ, Barratt DC: Automatic multi-organ segmentation on abdominal ct with dense v-networks. IEEE transactions on medical imaging 37(8), 1822– 1834 (2018) 29994628 PMC6076994
  3. [4]Hansen N, Müller SD, Koumoutsakos P: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation 11(1), 1– 18 (2003) 12804094
  4. [5]Hartley R, Zisserman A: Multiple view geometry in computer vision. Cambridge university press (2003)
  5. [7]Henderson P, Ferrari V: Learning single-image 3d reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision pp. 1– 20 (2019)
  6. [9]Jaderberg M, Simonyan K, Zisserman A,: Spatial transformer networks. In: Advances in neural information processing systems. pp. 2017– 2025 (2015)
  7. [10]Krčah M, Székely G, Blanc R: Fully automatic and fast segmentation of the femur bone from 3d-ct images with no shape prior. In: 2011 IEEE international symposium on biomedical imaging: from nano to macro pp. 2087– 2090. IEEE (2011)
  8. [11]Krebs J, Mansi T, Delingette H, Zhang L, Ghesu FC, Miao S, Maier AK, Ayache N, Liao R, Kamen A: Robust non-rigid registration through agent-based action learning In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 344– 352. Springer (2017)
  9. [12]Kuang D, Schmah T: Faim–a convnet method for unsupervised 3d medical image registration. In: International Workshop on Machine Learning in Medical Imaging pp. 646– 654. Springer (2019)
  10. [14]Liu S, Li T, Chen W, Li H: Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 7708– 7717 (2019)
  11. [15]Loper MM, Black MJ: Opendr: An approximate differentiable renderer In: European Conference on Computer Vision. pp. 154– 169. Springer (2014)
  12. [20]Miolane N, Mathe J, Donnat C, Jorda M, Pennec X: geomstats: a python package for riemannian geometry in machine learning. arXiv preprint arXiv:1805.08308 (2018) · arXiv
  13. [21]Penney GP, Weese J, Little JA, Desmedt P, Hill DL,: A comparison of similarity measures for use in 2-d-3-d medical image registration. IEEE transactions on medical imaging 17(4), 586– 595 (1998) 9845314
  14. [22]Roth HR, Lu L, Seff A, Cherry KM, Hoffman J, Wang S, Liu J, Turkbey E, Summers RM: A new 2.5 d representation for lymph node detection using random sets of deep convolutional neural network observations In: International conference on medical image computing and computer-assisted intervention. pp. 520– 527. Springer (2014) PMC4295635 25333158
  15. [23]Salehi SSM, Khan S, Erdogmus D, Gholipour A: Real-time deep registration with geodesic loss. arXiv preprint arXiv:1803.05982 (2018) · arXiv
  16. [24]Smith LN: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) pp. 464– 472. IEEE (2017)
  17. [25]Würfl T, Hoffmann M, Christlein V, Breininger K, Huang Y, Unberath M, Maier AK: Deep learning computed tomography: Learning projection-domain weights from image domain in limited angle problems. IEEE transactions on medical imaging 37(6), 1454– 1463 (2018) 29870373
  18. [26]Yan X, Yang J, Yumer E, Guo Y, Lee H: Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In: Advances in neural information processing systems. pp. 1696– 1704 (2016)

Cited by

Sorted by citations

Authors
Gao, C., Liu, X., Gu, W., Killeen, B., Armand, M., Taylor, R., & Unberath, M.
Affiliations
Johns Hopkins University (US)
Corresponding author
Gao, Cong
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Published
2020
Volume
12263
Pages
329-339
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-030-59716-0_32
PMC
PMC7602675
OpenAlex
W3092242645
Cited by
46
References
8
Citations by year
2026: 3 · 2025: 4 · 2024: 13 · 2023: 9 · 2022: 10 · 2021: 6 · 2020: 1
FWCI
5.78
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Institute of Biomedical Imaging and Bioengineering (R21EB020113, R01EB023939, R21EB028505)
Topics
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
spatial transformer network, projective geometry, differentiable rendering, 2D-3D registration, pose estimation, volumetric medical images

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review