Cited by 102Medical Image Segmentation TechniquesAnatomy and Medical TechnologySurgical Simulation and Training

The DOI matches its Crossref record; 117 of the 131 references it cites were found in a registry or an index; 33 of the 83 sources its claims cite were checked for support.

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Open access

Abstract

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review and perspective examines how machine learning has been applied to image-based 2D/3D registration for image-guided interventions, reviewing 48 studies published from January 2012 through February 2021. It finds promising improvements in capture range, representation learning, and direct pose regression, but persistent gaps in domain shift, multi-object registration, verification/uncertainty, and standardized evaluation.

Key findings

  • Across 48 included studies, most addressed X-ray to CT/CBCT registration (34), followed by X-ray to 3D object models (10); clinical applications included orthopedics (19), angiography (9), radiation therapy (7), and cephalometry (4).
  • The review identified five machine learning themes: contextualization (14 studies), representation learning (12), similarity modeling (10), direct parameter regression (22), and verification (4). Most methods were single-object: 45 of 48 studies considered only one object, and the few multiple-object methods solved objects independently.
  • Training data were often synthetic: 16 of 48 studies used real clinical or cadaveric data to train, while among 32 deep learning studies, 24 trained on synthetic data, 7 on real data, and 1 used a pretrained network. Seven studies comparing synthetic and real performance found substantial drops under domain shift, and ten synthetic-trained studies never meaningfully tested on real data.
  • The authors conclude that machine learning has improved capture range and enabled fast direct regression, but ambiguity, high-dimensional optimization, verification/uncertainty, multi-object and foreign-object handling, and generalizability remain open problems requiring standardized metrics, public benchmarks, and more generalist components.

Methods

  • PRISMA systematic review searched PubMed, embase, Business Source Ultimate, and Compendex for January 2012–February 2021 using 2D/3D registration terms combined with learning/training/testing terms; 559 records were found, 495 unique studies screened, 447 excluded, and 48 included for full-text review.
  • Five authors extracted data using a template covering method summary, modalities and registration phase, spatial transformation, machine learning model and training setup, evaluation strategy including domain shift, and methodological concerns; each article was coded by at least two authors, with one author merging reports into a consensus document.
  • The paper synthesizes findings narratively around five themes and a perspective discussion, without a quantitative meta-analysis.

Limitations

  • Stated in the paper: the search may not be exhaustive because it omitted newer or less popular terminology such as “pose regression” and modality-specific terms such as “video to CT registration”; it may have excluded studies not explicitly characterized as machine/deep learning or from the general computer vision literature; and it was limited to published manuscripts, so publication bias may have excluded relevant work.
  • Evident from the text: the review is qualitative and narrative rather than meta-analytic, and the included studies use heterogeneous metrics, datasets, and evaluation protocols, with many relying on synthetic data and few reporting real-data performance, which limits comparability and real-world conclusions; the authors also note that private code and datasets inhibit reproducibility.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/frobt.2021.716007

Cites

117 of 117 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

FIGURE 1 (A) A high-level overview of the workflow proposed by Grupp et al. (2019), which uses 2D/3D registration for estimating the relative pose of a periacetabular osteotomy (PAO) fragment. By enabling intra-operative 3D visualizations and the calculation of biomechanical measurements, this pose information should allow surgeons to better assess when a PAO fragment requires further adjustments and potentially reduce post-operative complications. The utility of the proposed workflow is diminished by the traditional registration strategy’s requirement for manual annotations, which are needed to initialize the pelvis pose and reconstruct the fragment shape. (B): Image-based navigation for robot-assisted femoropalsty by Gao et al. (2020a). The intra-operative poses of the robot and the femur anatomy are estimated using X-ray-image based 2D/3D registration. The robot-held drilling/injection device is positioned according to the pre-planned trajectory that is propageted intra-operatively using pose estimates from 2D/3D registration. Image-based navigation is less invasive than fiducial-based alternatives and simplifies the procedure.
FIGURE 1 (A) A high-level overview of the workflow proposed by Grupp et al. (2019), which uses 2D/3D registration for estimating the relative pose of a periacetabular osteotomy (PAO) fragment. By enabling intra-operative 3D visualizations and the calculation of biomechanical measurements, this pose information should allow surgeons to better assess when a PAO fragment requires further adjustments and potentially reduce post-operative complications. The utility of the proposed workflow is diminished by the traditional registration strategy’s requirement for manual annotations, which are needed to initialize the pelvis pose and reconstruct the fragment shape. (B): Image-based navigation for robot-assisted femoropalsty by Gao et al. (2020a). The intra-operative poses of the robot and the femur anatomy are estimated using X-ray-image based 2D/3D registration. The robot-held drilling/injection device is positioned according to the pre-planned trajectory that is propageted intra-operatively using pose estimates from 2D/3D registration. Image-based navigation is less invasive than fiducial-based alternatives and simplifies the procedure.CC BY · PubMed Central
FIGURE 2 PRISMA flow chart illustrating the screening and inclusion process. Duplicate studies were the result of searching multiple databases. Exclusion screening was performed individually with each study’s abstract with assistance of Covidence tool. Additional twelve studies excluded after full-text review, resulting in a pool of 48 studies included for full review.
FIGURE 2 PRISMA flow chart illustrating the screening and inclusion process. Duplicate studies were the result of searching multiple databases. Exclusion screening was performed individually with each study’s abstract with assistance of Covidence tool. Additional twelve studies excluded after full-text review, resulting in a pool of 48 studies included for full review.CC BY · PubMed Central
FIGURE 3 Illustrations of the main themes of machine learning in 2D/3D registration. The logic relationships of these themes are shown on top. We use a spine CT volume and a spine X-ray image as an example to show the generic 2D/3D projection geometry. Machine learning models are represented with a neural network icon. Key labels and parameters are presented and map to Eqs. 1, 2.
FIGURE 3 Illustrations of the main themes of machine learning in 2D/3D registration. The logic relationships of these themes are shown on top. We use a spine CT volume and a spine X-ray image as an example to show the generic 2D/3D projection geometry. Machine learning models are represented with a neural network icon. Key labels and parameters are presented and map to Eqs. 1, 2.CC BY · PubMed Central
FIGURE 4 Fluoroscopic images of chiseling performed during a cadaveric periacetabular osteotomy procedure. A cut along the ischium of the pelvis is shown by the oblqiue view in (A). Due to the difficulty with manually interpreting lateral orientations of the tool in an oblique view, the very next fluoroscopy frame was collected at an approximate anterior-posterior (AP) view, shown in (B). Although the osteotome location was confirmed by changing view points, the process of adjusting the C-arm by such a large offset can potentially increase operative time or cause the clinician to lose some context in the previous oblique view. Another oblique view is shown in (c), where the flat osteotome is used to complete the posterior cut, resulting in the creation of two bone fragments from the pelvis. In (C), the angled osteotome was left in the field of view and used as a visual aid for navigating the flat chisel and completing the osteotomy.
FIGURE 4 Fluoroscopic images of chiseling performed during a cadaveric periacetabular osteotomy procedure. A cut along the ischium of the pelvis is shown by the oblqiue view in (A). Due to the difficulty with manually interpreting lateral orientations of the tool in an oblique view, the very next fluoroscopy frame was collected at an approximate anterior-posterior (AP) view, shown in (B). Although the osteotome location was confirmed by changing view points, the process of adjusting the C-arm by such a large offset can potentially increase operative time or cause the clinician to lose some context in the previous oblique view. Another oblique view is shown in (c), where the flat osteotome is used to complete the posterior cut, resulting in the creation of two bone fragments from the pelvis. In (C), the angled osteotome was left in the field of view and used as a visual aid for navigating the flat chisel and completing the osteotomy.CC BY · PubMed Central

Verification

Claims the paper makes

88 Own interpretation · 63 No basis given · 40 Own result · 30 Not checked yet · 14 Partly supported · 12 No full text of the sources · 6 Sources not found · 5 Supported so far · 2 Mixed support · 1 Supported · 1 Not supported · 1 Own method · accuracy not yet measured

Show all 134 claims

  • It may state two separate claims as one. 82
  • Probably not a substantial claim. 76
  • The quote only partly states this claim. 1
129 of 129
  • The authors state this as an assumption.

    The reduction in dimensionality in 2D/3D registration originates from a projective, not an affine, transformation.

    “For the purposes of this review, we will assume that the reduction in dimensionality originates from a projective, not an affine, transformation.”
  • The authors state this as an assumption.

    The 2D view modality is X-ray.

    “Since the vast majority of studies examined concern 2D X-ray images, we also assume that 2D view modality is X-ray.”
  • The paper relies on this without support.

    New researchers typically spend a great deal of time implementing a traditional registration pipeline.

    “New researchers typically spend a great deal of time implementing a traditional registration pipeline so that traditional methods may be leveraged in conjunction with the development of learning-based approaches.”
  • The paper relies on this without support.

    2D/3D registration leads a niche existence.

    “2D/3D registration already leads a niche existence”
  • The paper relies on this without support.

    Machine learning models have a compact domain assumption that confines solutions to the data domain they were developed on.

    “the compact domain assumption of machine learning models that confine solutions to the data domain they were developed on.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 131 references

Research works printed but not linked: 6

  1. Amrehn, 2017, Ui-net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model, Proc. Eurographics Workshop Vis. Comput. Biol. Med., 143
  2. Chen, 2018, Real-time 2d/3d Registration of Vertebra via Machine Learning and Geometric Transformation, Zidonghua Xuebao/Acta Automatica Sinica, 44, 1183
  3. Miao, Real-time 2d/3d Registration via Cnn Regression, 1430
  4. 2021, Artificial Intelligence/machine Learning (Ai/ml)-based Software as a Medical Device (Samd) Action Plan
  5. Varnavas, Fully Automated 2d-3d Registration and Verification, Med. image Anal., 26, 108, 10.1016/j.media.2015.08.005 · DOI · Paper page
  6. Xiangqian, 2020, 2d/3d Medical Image Registration Using Convolutional Neural Network, Chin. J. Biomed. Eng., 39, 394

Cited by

Sorted by citations

See all 102

Metadata

Authors
Unberath, M., Gao, C., Hu, Y., Judish, M., Taylor, R. H., Armand, M., & Grupp, R.
Affiliations
Johns Hopkins University (US)
Venue
Frontiers in Robotics and AI
Published
2021-08-30
Volume
8
Pages
716007-716007
ISSN
2296-9144 (ISSN-L)
OpenAlex
W3190313291
Cited by
102
References
131
Citations by year
2026: 19 · 2025: 23 · 2024: 33 · 2023: 17 · 2022: 9 · 2021: 1
FWCI
2.14
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Robotics and AI (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
National Institute of Biomedical Imaging and Bioengineering (R21 EB028505)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Engineering › Biomedical Engineering › Anatomy and Medical Technology
  • Health Sciences › Medicine › Surgery › Surgical Simulation and Training
Keywords
2D-3D registration, image-guided interventions, machine learning, image-based navigation, minimally invasive surgery, fluoroscopy, endoscopy, surgical navigation, open problems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review