The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness
Abstract
Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to strictly aligned source images and cause severe artifacts in the fusion results when input images have slight shifts or deformations. In addition, the fusion results typically only have good visual effect, but neglect the semantic requirements of high-level vision tasks. This study incorporates image registration, image fusion, and semantic requirements of high-level vision tasks into a single framework and proposes a novel image registration and fusion method, named SuperFusion. Specifically, we design a registration network to estimate bidirectional deformation fields to rectify geometric distortions of input images under the supervision of both photometric and end-point constraints. The registration and fusion are combined in a symmetric scheme, in which while mutual promotion can be achieved by optimizing the naive fusion loss, it is further enhanced by the mono-modal consistent constraint on symmetric fusion outputs. In addition, the image fusion network is equipped with the global spatial attention mechanism to achieve adaptive feature integration. Moreover, the semantic constraint based on the pre-trained segmentation model and Lovasz-Softmax loss is deployed to guide the fusion network to focus more on the semantic requirements of high-level vision tasks. Extensive experiments on image registration, image fusion, and semantic segmentation tasks demonstrate the superiority of our SuperFusion compared to the state-of-the-art alternatives. The source code and pre-trained model are publicly available at https://github.com/Linfeng-Tang/SuperFusion.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Nonrigid registration using free-form deformations: application to breast mr images
Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D. L. G., Leach, M. O., & Hawkes, D. J. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.796284 · Cited by 5,265
FusionGAN: A generative adversarial network for infrared and visible image fusion
Ma, J., Yu, W., Liang, P., Li, C., & Jiang, J. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.09.004 · Cited by 2,267
Self-citationVoxelMorph: A Learning Framework for Deformable Medical Image Registration
Balakrishnan, G., Zhao, A., Sabuncu, M. R., Guttag, J., & Dalca, A. V. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2897538 · Cited by 2,174
U2Fusion: A Unified Unsupervised Image Fusion Network
Xu, H., Ma, J., Jiang, J., Guo, X., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3012548 · Cited by 2,127
Self-citationDenseFuse: A Fusion Approach to Infrared and Visible Images
Li, H., & Wu, X.-J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2887342 · Cited by 2,032
Infrared and visible image fusion methods and applications: A survey
Ma, J., Ma, Y., & Li, C. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.02.004 · Cited by 1,732
Self-citationIFCNN: A general image fusion framework based on convolutional neural network
Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X., & Zhang, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.07.011 · Cited by 1,550
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526
Self-citationMa, J., Xu, H., Jiang, J., Mei, X., & Zhang, X.-P. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2977573 · Cited by 1,343
Self-citationLiu, J., Fan, X., Huang, Z., Wu, G., Liu, R., Zhong, W., & Luo, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00571 · Cited by 1,336
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
Bell, S., Zitnick, C. L., Bala, K., & Girshick, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.314 · Cited by 1,292
RFN-Nest: An end-to-end residual fusion network for infrared and visible images
Li, H., Wu, X.-J., & Kittler, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.023 · Cited by 1,173
PIAFusion: A progressive infrared and visible image fusion network based on illumination aware
Tang, L., Yuan, J., Zhang, H., Jiang, X., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.007 · Cited by 1,161
Self-citationTang, L., Yuan, J., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.004 · Cited by 1,136
Self-citationInfrared and visible image fusion via gradient transfer and total variation minimization
Ma, J., Chen, C., Li, C., & Huang, J. (2016) · Information Fusion · doi:10.1016/j.inffus.2016.02.001 · Cited by 1,128
Self-citationFast Normalized Cross-Correlation
Yoo, J.-C., & Han, T. H. (2009) · Circuits, Systems and Signal Processing · doi:10.1007/s00034-009-9130-7 · Cited by 1,115
Image Fusion With Convolutional Sparse Representation
Liu, Y., Chen, X., Ward, R. K., & Jane Wang, Z. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2618776 · Cited by 986
Image fusion meets deep learning: A survey and perspective
Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.008 · Cited by 834
Self-citationLi, H., Wu, X.-J., & Durrani, T. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3005230 · Cited by 798
Ma, J., Zhou, Z., Wang, B., & Zong, H. (2017) · Infrared Physics & Technology · doi:10.1016/j.infrared.2017.02.005 · Cited by 773
Self-citationSDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion
Zhang, H., & Ma, J. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01501-8 · Cited by 747
Self-citationMFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693
Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset
Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., & Lau, R. W. H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01255 · Cited by 681
RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform
Li, J., Hu, Q., & Ai, M. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2959244 · Cited by 651
MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion
Li, H., Wu, X.-J., & Kittler, J. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2975984 · Cited by 627
Ma, J., Zhang, H., Shao, Z., Liang, P., & Xu, H. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3038013 · Cited by 618
Self-citationZhang, H., Xu, H., Xiao, Y., Guo, X., & Ma, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6975 · Cited by 589
Self-citationFusionDN: A Unified Densely Connected Network for Image Fusion
Xu, H., Ma, J., Le, Z., Jiang, J., & Guo, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6936 · Cited by 587
Self-citationUnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss
Meister, S., Hur, J., & Roth, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12276 · Cited by 520
YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer
Tang, W., He, F., & Liu, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3192661 · Cited by 485
STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection
Ma, J., Tang, L., Xu, M., Zhang, H., & Xiao, G. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3075747 · Cited by 483
Self-citationInfrared and visible image fusion via detail preserving adversarial learning
Ma, J., Liang, P., Yu, W., Chen, C., Guo, X., Wu, J., & Jiang, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.07.005 · Cited by 456
Self-citationInfrared and visible image fusion based on target-enhanced multiscale transform decomposition
Chen, J., Li, X., Luo, L., Mei, X., & Ma, J. (2020) · Information Sciences · doi:10.1016/j.ins.2019.08.066 · Cited by 453
Self-citationMedical image registration using mutual information
Maes, F., Vandermeulen, D., & Suetens, P. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817864 · Cited by 432
Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion
Liu, J., Fan, X., Jiang, J., Liu, R., & Luo, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3056725 · Cited by 431
Li, J., Huo, H., Li, C., Wang, R., & Feng, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2997127 · Cited by 415
DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion
Zhao, Z., Xu, S., Zhang, C., Liu, J., Zhang, J., & Li, P. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/135 · Cited by 341
Wang, D., Liu, J., Fan, X., & Liu, R. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/487 · Cited by 318
Classification Saliency-Based Rule for Visible and Infrared Image Fusion
Xu, H., Zhang, H., & Ma, J. (2021) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2021.3100986 · Cited by 294
Self-citationA deep learning framework for remote sensing image registration
Wang, S., Quan, D., Liang, X., Ning, M., Guo, Y., & Jiao, L. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.12.012 · Cited by 292
VIFB: A Visible and Infrared Image Fusion Benchmark
Zhang, X., Ye, P., & Xiao, G. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00060 · Cited by 250
Direction-Aware Spatial Context Features for Shadow Detection and Removal
Hu, X., Fu, C.-W., Zhu, L., Qin, J., & Heng, P.-A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2919616 · Cited by 242
RXDNFuse: A aggregated residual dense network for infrared and visible image fusion
Long, Y., Jia, H., Zhong, Y., Jiang, Y., & Jia, Y. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.11.009 · Cited by 233
Zhang, H., Yuan, J., Tian, X., & Ma, J. (2021) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2021.3119954 · Cited by 215
Self-citationObject fusion tracking based on visible and infrared images: A comprehensive review
Zhang, X., Ye, P., Leung, H., Gong, K., & Xiao, G. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.05.002 · Cited by 208
GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences
Truong, P., Danelljan, M., & Timofte, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00629 · Cited by 206
Deep Learning-based Multi-focus Image Fusion: A Survey and A Comparative Study
Zhang, X. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3078906 · Cited by 206
Vs, V., Jose Valanarasu, J. M., Oza, P., & Patel, V. M. (2022) · 2022 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip46576.2022.9897280 · Cited by 205
DRF: Disentangled Representation for Visible and Infrared Image Fusion
Xu, H., Wang, X., & Ma, J. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3056645 · Cited by 187
Self-citationRFNet: Unsupervised Network for Mutually Reinforcing Multi-modal Image Registration and Fusion
Xu, H., Ma, J., Yuan, J., Le, Z., & Liu, W. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01906 · Cited by 187
Self-citationZhou, H., Wu, W., Zhang, Y., Ma, J., & Ling, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3129609 · Cited by 185
Self-citationInfrared and Visible Image Fusion via Texture Conditional Generative Adversarial Network
Yang, Y., Liu, J., Huang, S., Wan, W., Wen, W., & Guan, J. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3054584 · Cited by 176
Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation
Arar, M., Ginger, Y., Danon, D., Bermano, A. H., & Cohen-Or, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01342 · Cited by 175
CGTF: Convolution-Guided Transformer for Infrared and Visible Image Fusion
Li, J., Zhu, J., Li, C., Chen, X., & Yang, B. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3175055 · Cited by 171
Multispectral Image Feature Points
Aguilera, C., Barrera, F., Lumbreras, F., Sappa, A. D., & Toledo, R. (2012) · Sensors · doi:10.3390/s120912661 · Cited by 168
Infrared and visible images fusion based on RPCA and NSCT
Fu, Z., Wang, X., Xu, J., Zhou, N., & Zhao, Y. (2016) · Infrared Physics & Technology · doi:10.1016/j.infrared.2016.05.012 · Cited by 159
Multisensor Image Fusion and Enhancement in Spectral Total Variation Domain
Zhao, W., Lu, H., & Wang, D. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2760100 · Cited by 148
Self-supervised feature adaption for infrared and visible image fusion
Zhao, F., Zhao, W., Yao, L., & Liu, Y. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.002 · Cited by 101
PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image FusionPreprint
Fu, Y., Xu, T., Wu, X., & Kittler, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.13967 · Cited by 96
SF-Net: A Multi-Task Model for Brain Tumor Segmentation in Multimodal MRI via Image Fusion
Liu, Y., Mu, F., Shi, Y., & Chen, X. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3198594 · Cited by 94
Searching a Hierarchically Aggregated Fusion Architecture for Fast Multi-Modality Image Fusion
Liu, R., Liu, Z., Liu, J., & Fan, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475299 · Cited by 87
Glioma Segmentation-Oriented Multi-Modal MR Image Fusion With Adversarial Learning
Liu, Y., Shi, Y., Mu, F., Cheng, J., & Chen, X. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105770 · Cited by 87
Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks
Liu, Y., Shi, Y., Mu, F., Cheng, J., Li, C., & Chen, X. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3184360 · Cited by 66
DNDT: Infrared and Visible Image Fusion Via DenseNet and Dual-Transformer
Zhao, H., & Nie, R. (2021) · International Conference on Information Technology and Biomedical Engineering (ICITBE) · doi:10.1109/icitbe54178.2021.00025 · Cited by 64
DASC: Robust Dense Descriptor for Multi-Modal and Multi-Spectral Correspondence Estimation
Kim, S., Min, D., Ham, B., Do, M. N., & Sohn, K. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2615619 · Cited by 63
Promoting Single-Modal Optical Flow Network for Diverse Cross-Modal Flow Estimation
Zhou, S., Tan, W., & Yan, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20268 · Cited by 59
Infrared and visible image fusion via parallel scene and texture learning
Xu, M., Tang, L., Zhang, H., & Ma, J. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108929 · Cited by 59
Self-citationLearning Specific and General Realm Feature Representations for Image Fusion
Zhao, F., & Zhao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3016123 · Cited by 40
Cited by
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Xu, S., Lin, Z., Timofte, R., & Van Gool, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00572 · Cited by 948
CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach
Li, H., & Wu, X.-J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102147 · Cited by 460
Visible and Infrared Image Fusion Using Deep Learning
Zhang, X., & Demiris, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261282 · Cited by 392
Tang, L., Zhang, H., Xu, H., & Ma, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101870 · Cited by 385
Self-citationDif-Fusion: Toward High Color Fidelity in Infrared and Visible Image Fusion With Diffusion Models
Yue, J., Fang, L., Xia, S., Deng, Y., & Ma, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3322046 · Cited by 304
Self-citation
Metadata
- Authors
- Tang, L., Deng, Y., Ma, Y., Huang, J., & Ma, J.
- Venue
- IEEE/CAA Journal of Automatica Sinica
- Published
- 2022-12
- Volume
- 9
- Issue
- 12
- Pages
- 2121-2137
- ISSN
- 2329-9266 (ISSN-L)
- 2329-9274
- OpenAlex
- W4313267411
- Cited by
- 488
- References
- 71
- FWCI
- 36.40
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.