The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer
Abstract
Owing to the limitations of imaging sensors, it is challenging to obtain a medical image that simultaneously contains functional metabolic information and structural tissue details. Multimodal medical image fusion, an effective way to merge the complementary information in different modalities, has become a significant technique to facilitate clinical diagnosis and surgical navigation. With powerful feature representation ability, deep learning (DL)-based methods have improved such fusion results but still have not achieved satisfactory performance. Specifically, existing DL-based methods generally depend on convolutional operations, which can well extract local patterns but have limited capability in preserving global context information. To compensate for this defect and achieve accurate fusion, we propose a novel unsupervised method to fuse multimodal medical images via a multiscale adaptive Transformer termed MATR. In the proposed method, instead of directly employing vanilla convolution, we introduce an adaptive convolution for adaptively modulating the convolutional kernel based on the global complementary context. To further model long-range dependencies, an adaptive Transformer is employed to enhance the global semantic extraction capability. Our network architecture is designed in a multiscale fashion so that useful multimodal information can be adequately acquired from the perspective of different scales. Moreover, an objective function composed of a structural loss and a region mutual information loss is devised to construct constraints for information preservation at both the structural-level and the feature-level. Extensive experiments on a mainstream database demonstrate that the proposed method outperforms other representative and state-of-the-art methods in terms of both visual quality and quantitative evaluation. We also extend the proposed method to address other biomedical image fusion issues, and the pleasing fusion results illustrate that MATR has good generalization capability. The code of the proposed method is available at https://github.com/tthinking/MATR.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Liang, Y., He, F., Zeng, X., & Luo, J. (2021) · Integrated Computer-Aided Engineering · doi:10.3233/ica-210661 · Cited by 94
Self-citationRetractedExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
SwinIR: Image Restoration Using Swin Transformer
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., & Timofte, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00210 · Cited by 4,734
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206
Pre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308
U2Fusion: A Unified Unsupervised Image Fusion Network
Xu, H., Ma, J., Jiang, J., Guo, X., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3012548 · Cited by 2,127
Objective image fusion performance measure
Xydeas, C. S., & Petrović, V. (2000) · Electronics Letters · doi:10.1049/el:20000267 · Cited by 1,996
A general framework for image fusion based on multi-scale transform and sparse representation
Liu, Y., Liu, S., & Wang, Z. (2015) · Information Fusion · doi:10.1016/j.inffus.2014.09.004 · Cited by 1,431
Self-citationColor balance and fusion for underwater image enhancement
Ancuti, C. O., Ancuti, C., De Vleeschouwer, C., & Bekaert, P. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2759252 · Cited by 1,357
Ma, J., Xu, H., Jiang, J., Mei, X., & Zhang, X.-P. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2977573 · Cited by 1,343
Top-down influences on visual processing
Gilbert, C. D., & Li, W. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3476 · Cited by 1,224
A new image fusion performance metric based on visual information fidelity
Han, Y., Cai, Y., Cao, Y., & Xu, X. (2013) · Information Fusion · doi:10.1016/j.inffus.2011.08.002 · Cited by 1,071
Pixel- and region-based image fusion with complex wavelets
Lewis, J. J., O’Callaghan, R. J., Nikolov, S. G., Bull, D. R., & Canagarajah, N. (2007) · Information Fusion · doi:10.1016/j.inffus.2005.09.006 · Cited by 834
Image fusion meets deep learning: A survey and perspective
Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.008 · Cited by 834
A general framework for multiresolution image fusion: from pixels to regions
Piella, G. (2003) · Information Fusion · doi:10.1016/s1566-2535(03)00046-0 · Cited by 799
Deep learning for pixel-level image fusion: Recent advances and future prospects
Liu, Y., Chen, X., Wang, Z., Wang, Z. J., Ward, R. K., & Wang, X. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.007 · Cited by 736
Self-citationYin, M., Liu, X., Liu, Y., & Chen, X. (2019) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2018.2838778 · Cited by 594
Self-citationZhang, H., Xu, H., Xiao, Y., Guo, X., & Ma, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6975 · Cited by 589
Drzezga, A., Souvatzoglou, M., Eiber, M., Beer, A. J., Fürst, S., Martinez-Möller, A., Nekolla, S. G., Ziegler, S., Ganter, C., Rummeny, E. J., et al. (2012) · Journal of Nuclear Medicine · doi:10.2967/jnumed.111.098608 · Cited by 508
Comments on ‘Information measure for performance of image fusion’
Hossny, M., Nahavandi, S., & Creighton, D. (2008) · Electronics Letters · doi:10.1049/el:20081754 · Cited by 505
Zhou, Z., Wang, B., Li, S., & Dong, M. (2016) · Information Fusion · doi:10.1016/j.inffus.2015.11.003 · Cited by 475
Perceptual learning and top-down influences in primary visual cortex
Li, W., Piëch, V., & Gilbert, C. D. (2004) · Nature Neuroscience · doi:10.1038/nn1255 · Cited by 473
A medical image fusion method based on convolutional neural networks
Liu, Y., Chen, X., Cheng, J., & Peng, H. (2017) · 2017 20th International Conference on Information Fusion (Fusion) · doi:10.23919/icif.2017.8009769 · Cited by 391
Self-citationThe type VI secretion system of Vibrio cholerae fosters horizontal gene transfer
Borgeaud, S., Metzger, L. C., Scrignari, T., & Blokesch, M. (2015) · Science · doi:10.1126/science.1260064 · Cited by 389
Group-Sparse Representation With Dictionary Learning for Medical Image Denoising and Fusion
Li, S., Yin, H., & Fang, L. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2012.2217493 · Cited by 385
EMFusion: An unsupervised enhanced medical image fusion network
Xu, H., & Ma, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.001 · Cited by 315
PET/MRI for Neurologic Applications
Catana, C., Drzezga, A., Heiss, W.-D., & Rosen, B. R. (2012) · Journal of Nuclear Medicine · doi:10.2967/jnumed.112.105346 · Cited by 288
A human perception inspired quality metric for image fusion based on regional information
Chen, H., & Varshney, P. K. (2007) · Information Fusion · doi:10.1016/j.inffus.2005.10.001 · Cited by 286
MHF-Net: An Interpretable Deep Network for Multispectral and Hyperspectral Image Fusion
Xie, Q., Zhou, M., Zhao, Q., Xu, Z., & Meng, D. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3015691 · Cited by 268
Co-Learning Feature Fusion Maps From PET-CT Images of Lung Cancer
Kumar, A., Fulham, M., Feng, D., & Kim, J. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2923601 · Cited by 238
Image fusion metric based on mutual information and Tsallis entropy
Cvejic, N., Canagarajah, C. N., & Bull, D. R. (2006) · Electronics Letters · doi:10.1049/el:20060693 · Cited by 230
Koroleva, O. A., Tomlinson, M. L., Leader, D., Shaw, P., & Doonan, J. H. (2005) · The Plant Journal · doi:10.1111/j.1365-313x.2004.02281.x · Cited by 219
Zhao, J., & Liu, Z. (2007) · Cited by 219
Anatomical-Functional Image Fusion by Information of Interest in Local Laplacian Filtering Domain
Du, J., Li, W., & Xiao, B. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2745202 · Cited by 194
A Bilevel Integrated Model With Data-Driven Layer Ensemble for Multi-Modality Image Fusion
Liu, R., Liu, J., Jiang, Z., Fan, X., & Luo, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3043125 · Cited by 165
SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images
Wang, Z., Chen, Y., Shao, W., Li, H., & Zhang, L. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3191664 · Cited by 162
DRCDN: learning deep residual convolutional dehazing networks
Zhang, S., & He, F. (2020) · The Visual Computer · doi:10.1007/s00371-019-01774-8 · Cited by 158
Self-citationAIPNet: Image-to-Image Single Image Dehazing With Atmospheric Illumination Prior
Wang, A., Wang, W., Liu, J., & Gu, N. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2868567 · Cited by 155
Hybrid Contrastive Learning for Unsupervised Person Re-Identification
Si, T., He, F., Zhang, Z., & Duan, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3174414 · Cited by 118
Self-citationPerceptual Image Fusion Using Wavelets
Hill, P., Al-Mualla, M. E., & Bull, D. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2633863 · Cited by 110
Region Mutual Information Loss for Semantic SegmentationPreprint
Zhao, S., Wang, Y., Yang, Z., & Cai, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12037 · Cited by 100
Li, H., He, F., Chen, Y., & Pan, Y. (2021) · Memetic Computing · doi:10.1007/s12293-021-00328-7 · Cited by 100
Self-citationLin, X., Ma, L., Liu, W., & Chang, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_41 · Cited by 89
Poisson image fusion based on Markov random field fusion model
Sun, J. Z., Zhu, H., Xu, Z., & Han, C. (2012) · Information Fusion · doi:10.1016/j.inffus.2012.07.003 · Cited by 82
A New Multiwavelet-Based Approach to Image Fusion
Wang, H.-H. (2004) · Journal of Mathematical Imaging and Vision · doi:10.1023/b:jmiv.0000035181.00093.e3 · Cited by 71
Green Fluorescent Protein and Phase Contrast Image Fusion Via Detail Preserving Cross Network
Tang, W., Liu, Y., Cheng, J., Li, C., & Chen, X. (2021) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2021.3083965 · Cited by 47
Self-citationSimilarity-based multimodality image fusion with shiftable complex directional pyramid
Zhang, Q., Wang, L., Li, H., et al. (2011) · Pattern Recognition Letters · doi:10.1016/j.patrec.2011.06.002 · Cited by 41
Green Fluorescent Protein and Phase-Contrast Image Fusion via Generative Adversarial Networks
Tang, W., Liu, Y., Zhang, C., Cheng, J., Peng, H., & Chen, X. (2019) · Computational and Mathematical Methods in Medicine · doi:10.1155/2019/5450373 · Cited by 31
Self-citationTang, W., Liu, Y., Cheng, J., Li, C., Peng, H., & Chen, X. (2020) · Microscopy Research and Technique · doi:10.1002/jemt.23514 · Cited by 24
Self-citationTwo‐scale image decomposition based image fusion using structure tensor
Du, J., & Li, W. (2019) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22367 · Cited by 18
Green Fluorescent Protein and Phase Contrast Image Fusion via Dual Attention Residual Network
Tang, W., Wang, L., & Liu, Y. (2021) · IEEE International Conference on Medical Imaging Physics and Engineering (ICMIPE) · doi:10.1109/icmipe53131.2021.9698938 · Cited by 10
Self-citation
Cited by
DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer
Tang, W., He, F., Liu, Y., Duan, Y., & Si, T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3234340 · Cited by 571
Self-citationCrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach
Li, H., & Wu, X.-J. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102147 · Cited by 460
Visible and Infrared Image Fusion Using Deep Learning
Zhang, X., & Demiris, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261282 · Cited by 392
Metadata
- Authors
- Tang, W., He, F., Liu, Y., & Duan, Y.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2022
- Volume
- 31
- Pages
- 5134-5149
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W4288391270
- Cited by
- 442
- References
- 57
- FWCI
- 32.23
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.