The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion
Abstract
Existing image fusion methods are typically limited to aligned source images and have to "tolerate" parallaxes when images are unaligned. Simultaneously, the large variances between different modalities pose a significant challenge for multi-modal image registration. This study proposes a novel method called MURF, where for the first time, image registration and fusion are mutually reinforced rather than being treated as separate issues. MURF leverages three modules: shared information extraction module (SIEM), multi-scale coarse registration module (MCRM), and fine registration and fusion module (F2M). The registration is carried out in a coarse-to-fine manner. During coarse registration, SIEM first transforms multi-modal images into mono-modal shared information to eliminate the modal variances. Then, MCRM progressively corrects the global rigid parallaxes. Subsequently, fine registration to repair local non-rigid offsets and image fusion are uniformly implemented in F2M. The fused image provides feedback to improve registration accuracy, and the improved registration result further improves the fusion result. For image fusion, rather than solely preserving the original source information in existing methods, we attempt to incorporate texture enhancement into image fusion. We test on four types of multi-modal data (RGB-IR, RGB-NIR, PET-MRI, and CT-MRI). Extensive registration and fusion results validate the superiority and universality of MURF.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2022) · arXiv · doi:10.48550/arxiv.2207.02696 · Cited by 13,205
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00721 · Cited by 13,198
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 1×
“values for positive pairs while low values for negative ones, the contrastive loss function for learning f cl θ1(·) and f cl θ2(·) is defined as the InfoNCE loss [44]:”
BackgroundAVANTS, B., EPSTEIN, C., GROSSMAN, M., & GEE, J. (2008) · Medical Image Analysis · doi:10.1016/j.media.2007.06.004 · Cited by 5,669
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076
FusionGAN: A generative adversarial network for infrared and visible image fusion
Ma, J., Yu, W., Liang, P., Li, C., & Jiang, J. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.09.004 · Cited by 2,267
Self-citationAn overlap invariant entropy measure of 3D medical image alignment
Studholme, C., Hill, D. L. G., & Hawkes, D. J. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00091-0 · Cited by 2,180
U2Fusion: A Unified Unsupervised Image Fusion Network
Xu, H., Ma, J., Jiang, J., Guo, X., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3012548 · Cited by 2,127
Self-citationDenseFuse: A Fusion Approach to Infrared and Visible Images
Li, H., & Wu, X.-J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2887342 · Cited by 2,032
Infrared and visible image fusion methods and applications: A survey
Ma, J., Ma, Y., & Li, C. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.02.004 · Cited by 1,732
Self-citationIFCNN: A general image fusion framework based on convolutional neural network
Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X., & Zhang, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.07.011 · Cited by 1,550
A general framework for image fusion based on multi-scale transform and sparse representation
Liu, Y., Liu, S., & Wang, Z. (2015) · Information Fusion · doi:10.1016/j.inffus.2014.09.004 · Cited by 1,431
Ma, J., Xu, H., Jiang, J., Mei, X., & Zhang, X.-P. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2977573 · Cited by 1,343
Self-citationRFN-Nest: An end-to-end residual fusion network for infrared and visible images
Li, H., Wu, X.-J., & Kittler, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.023 · Cited by 1,173
PIAFusion: A progressive infrared and visible image fusion network based on illumination aware
Tang, L., Yuan, J., Zhang, H., Jiang, X., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.007 · Cited by 1,161
Self-citationTang, L., Yuan, J., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.004 · Cited by 1,136
Self-citationInfrared and visible image fusion via gradient transfer and total variation minimization
Ma, J., Chen, C., Li, C., & Huang, J. (2016) · Information Fusion · doi:10.1016/j.inffus.2016.02.001 · Cited by 1,128
Self-citationUnnatural L0 Sparse Representation for Natural Image Deblurring
Xu, L., Zheng, S., & Jia, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.147 · Cited by 1,079
A new image fusion performance metric based on visual information fidelity
Han, Y., Cai, Y., Cao, Y., & Xu, X. (2013) · Information Fusion · doi:10.1016/j.inffus.2011.08.002 · Cited by 1,071
WLD: A Robust Local Image Descriptor
Chen, J., Shan, S., He, C., Zhao, G., Pietikäinen, M., Chen, X., & Gao, W. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.155 · Cited by 1,024
Image Matching from Handcrafted to Deep Features: A Survey
Ma, J., Jiang, X., Fan, A., Jiang, J., & Yan, J. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01359-2 · Cited by 990
Self-citationImage fusion meets deep learning: A survey and perspective
Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.008 · Cited by 834
Self-citationLi, H., Wu, X.-J., & Durrani, T. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3005230 · Cited by 798
Ma, J., Zhou, Z., Wang, B., & Zong, H. (2017) · Infrared Physics & Technology · doi:10.1016/j.infrared.2017.02.005 · Cited by 773
Self-citationLiu, Z., Blasch, E., Xue, Z., Zhao, J., Laganiere, R., & Wu, W. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.109 · Cited by 717
Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation
Wei, Q., Bioucas-Dias, J., Dobigeon, N., & Tourneret, J.-Y. (2015) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2014.2381272 · Cited by 698
MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion
Li, H., Wu, X.-J., & Kittler, J. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2975984 · Cited by 627
Ma, J., Zhang, H., Shao, Z., Liang, P., & Xu, H. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3038013 · Cited by 618
Self-citationA review of multimodal image matching: Methods and applications
Jiang, X., Ma, J., Xiao, G., Shao, Z., & Guo, X. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.012 · Cited by 532
Self-citationInfrared and visible image fusion via detail preserving adversarial learning
Ma, J., Liang, P., Yu, W., Chen, C., Guo, X., Wu, J., & Jiang, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.07.005 · Cited by 456
Self-citationA fast normalized cross-correlation calculation method for motion estimation
Luo, J., & Konofagou, E. E. (2010) · IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control · doi:10.1109/tuffc.2010.1554 · Cited by 384
EMFusion: An unsupervised enhanced medical image fusion network
Xu, H., & Ma, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.001 · Cited by 315
Self-citationNonrigid Image Registration Using Conditional Mutual Information
Loeckx, D., Slagmolen, P., Maes, F., Vandermeulen, D., & Suetens, P. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2021843 · Cited by 257
Li, G., Lin, Y., & Qu, X. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.008 · Cited by 243
RXDNFuse: A aggregated residual dense network for infrared and visible image fusion
Long, Y., Jia, H., Zhong, Y., Jiang, Y., & Jia, Y. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.11.009 · Cited by 233
Image Matching by Normalized Cross-Correlation
Zhao, F., Huang, Q., & Gao, W. (2006) · 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings · doi:10.1109/icassp.2006.1660446 · Cited by 215
Image fusion using hybrid methods in multimodality medical images
Yadav, S. P., & Yadav, S. (2020) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-020-02136-6 · Cited by 201
Huang, Z., Lv, C., Xing, Y., & Wu, J. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3003121 · Cited by 188
DRF: Disentangled Representation for Visible and Infrared Image Fusion
Xu, H., Wang, X., & Ma, J. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3056645 · Cited by 187
Self-citationRFNet: Unsupervised Network for Mutually Reinforcing Multi-modal Image Registration and Fusion
Xu, H., Ma, J., Yuan, J., Le, Z., & Liu, W. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01906 · Cited by 187
Self-citationUnsupervised Deep Image Fusion With Structure Tensor Representations
Jung, H., Kim, Y., Jang, H., Ha, N., & Sohn, K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2966075 · Cited by 186
Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation
Arar, M., Ginger, Y., Danon, D., Bermano, A. H., & Cohen-Or, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01342 · Cited by 175
Studholme, C., Drapaca, C., Iordanova, B., & Cardenas, V. (2006) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2006.872745 · Cited by 164
CoMIR: Contrastive Multimodal Image Representation for RegistrationPreprint
Pielawski, N., Wetzer, E., Öfverstedt, J., Lu, J., Wählby, C., Lindblad, J., & Sladoje, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06325 · Cited by 121
DASC: Dense adaptive self-correlation descriptor for multi-modal and multi-spectral correspondence
Kim, S., Min, D., Ham, B., Ryu, S., Do, M. N., & Sohn, K. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298822 · Cited by 97
Boosting Structure Consistency for Multispectral and Multimodal Image Registration
Cao, S.-Y., Shen, H., Chen, S., & Li, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2980972 · Cited by 71
Normalized Total Gradient: A New Measure for Multispectral Image Registration
Chen, S.-J., Shen, H., Li, C., & Xin, J. H. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2776753 · Cited by 65
DASC: Robust Dense Descriptor for Multi-Modal and Multi-Spectral Correspondence Estimation
Kim, S., Min, D., Ham, B., Do, M. N., & Sohn, K. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2615619 · Cited by 63
Hyperspectral-Multispectral Image Fusion via Tensor Ring and Subspace Decompositions
Xu, H., Qin, M., Chen, S., Zheng, Y., & Zheng, J. W. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3108233 · Cited by 62
Self-citationEl-Mezouar, M. C., Kpalma, K., Taleb, N. C.-E., et al. (2014) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2014.2306332 · Cited by 59
Fusing Infrared and Visible Images of Different Resolutions via Total Variation Model
Du, Q., Xu, H., Ma, Y., et al. (2018) · Sensors · doi:10.3390/s18113827 · Cited by 53
Self-citationImage fusion with saliency map and interest points
Meng, F., Guo, B., Song, M., & Zhang, X. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.10.080 · Cited by 43
Structural image representation for image registration
Wachinger, C., & Navab, N. (2010) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2010.5543432 · Cited by 27
FIRE: Unsupervised bi-directional inter-modality registration using deep networksPreprint
Wang, C., Papanastasiou, G., Chartsias, A., Jacenków, G., Tsaftaris, S. A., & Zhang, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05062 · Cited by 16
LaHaye, N., Garay, M. J., Bue, B., El-Askary, H. M., & Linstead, E. J. (2021) · Remote Sensing · doi:10.3390/rs13122364 · Cited by 9
Cited by
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion
Zhao, Z., Bai, H., Zhu, Y., Zhang, J., Xu, S., Zhang, Y., Zhang, K., Meng, D., Timofte, R., & Van Gool, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00742 · Cited by 431
Metadata
- Authors
- Xu, H., Yuan, J., & Ma, J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-10
- Volume
- 45
- Issue
- 10
- Pages
- 12148-12166
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4379741379
- Cited by
- 288
- References
- 57
- FWCI
- 31.80
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.