FusionMamba: dynamic feature enhancement for multimodal image fusion with Mamba
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1×
“Mamba’s versatility and efficiency make it a compelling choice for a wide range of applications, from language processing to computer vision representation tasks [6, 18].”
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01155 · Cited by 9,641 · cited 1×
“2(g)) [28] into DVSS.”
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Proceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
FusionGAN: A generative adversarial network for infrared and visible image fusion
Ma, J., Yu, W., Liang, P., Li, C., & Jiang, J. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.09.004 · Cited by 2,267 · cited 4×
“…learning, the primary image fusion methodologies can be categorized into four groups: Convolutional Neural Network (CNN) models [30, 31, 37], Generative Ad-versarial Network (GAN) models [16, 17], Autoencoder (AE) based models [10–12, 20] , and models based on Transformer architectures [ 15, 24”
InfluentialProceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
U2Fusion: A Unified Unsupervised Image Fusion Network
Xu, H., Ma, J., Jiang, J., Guo, X., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3012548 · Cited by 2,127
DenseFuse: A Fusion Approach to Infrared and Visible Images
Li, H., & Wu, X.-J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2887342 · Cited by 2,032 · cited 8×
“These hybrid enhanced features are then input into the Efficient Spatial Scanning 2D (ES2D) [ ?”
InfluentialProceedings of the IEEE International Conference on Image Processing
Calway, A. D. (1996) · The Institute of Electrical and Electronics Engineers · Cited by 1,578
IFCNN: A general image fusion framework based on convolutional neural network
Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X., & Zhang, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.07.011 · Cited by 1,550
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526 · cited 10×
“Although certain fusion models [15, 27, 29, 38] adopt a hybrid approach by amalgamating convolutional and Transformer layers to leverage their respective strengths and mitigate weaknesses, computational overhead remains a significant concern.”
InfluentialMa, J., Xu, H., Jiang, J., Mei, X., & Zhang, X.-P. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2977573 · Cited by 1,343 · cited 8×
“SwinFusion [15] achieves satisfactory fusion but with excessively sharp edges.”
InfluentialProceedings of the British Machine Vision Conference 2014
(2014) · doi:10.5244/c.28 · Cited by 1,337
Multi-focus image fusion with a deep convolutional neural network
Liu, Y., Chen, X., Peng, H., & Wang, Z. (2017) · Information Fusion · doi:10.1016/j.inffus.2016.12.001 · Cited by 1,327 · cited 1×
“After that, the discretization process is typically adopted into practical deep learning algorithms.”
Multispectral pedestrian detection: Benchmark dataset and baseline
Hwang, S., Park, J., Kim, N., Choi, Y., & Kweon, I. S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298706 · Cited by 1,249 · cited 5×
“We compare the CNN-based U2Fusion [30], IFCNN [37] method with the Transformer-based IFT [27] and SwinFusion [15] method.”
InfluentialRFN-Nest: An end-to-end residual fusion network for infrared and visible images
Li, H., Wu, X.-J., & Kittler, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.02.023 · Cited by 1,173 · cited 2×
“…learning, the primary image fusion methodologies can be categorized into four groups: Convolutional Neural Network (CNN) models [30, 31, 37], Generative Ad-versarial Network (GAN) models [16, 17], Autoencoder (AE) based models [10–12, 20] , and models based on Transformer architectures [ 15, 24”
PIAFusion: A progressive infrared and visible image fusion network based on illumination aware
Tang, L., Yuan, J., Zhang, H., Jiang, X., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.007 · Cited by 1,161 · cited 1×
“Expanding beyond language tasks, Mamba has also been applied successfully to vision tasks like image classification, video understanding, and biomedical image segmentation.”
Tang, L., Yuan, J., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.004 · Cited by 1,136 · cited 1×
“Expanding beyond language tasks, Mamba has also been applied successfully to vision tasks like image classification, video understanding, and biomedical image segmentation.”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space LayersPreprint
Gu, A., Johnson, I., Goel, K., Saab, K. K., Dao, T., Rudra, A., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13985 · Cited by 1,050
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Xu, S., Lin, Z., Timofte, R., & Van Gool, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00572 · Cited by 948 · cited 1×
“Although certain fusion models [15, 27, 29, 38] adopt a hybrid approach by amalgamating convolutional and Transformer layers to leverage their respective strengths and mitigate weaknesses, computational overhead remains a significant concern.”
Li, H., Wu, X.-J., & Durrani, T. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3005230 · Cited by 798 · cited 8×
“6, FusionGAN and U2Fusion [30] exhibit unsatisfactory brightness and clarity, with some gray matter blurring the texture details.”
InfluentialSDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion
Zhang, H., & Ma, J. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01501-8 · Cited by 747 · cited 6×
“7 and Table 4 illustrate that RFN-Nest[12], DATFuse [25], SDNet [35], and U2Fusion [30] struggle to accurately represent scene information in visible images due to inadequate global information interaction and improper intensity con-trol.”
InfluentialMambaIR: A Simple Baseline for Image Restoration with State-Space Model
Guo, H., Li, J., Dai, T., Ouyang, Z., Ren, X., & Xia, S.-T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72649-1_13 · Cited by 642
Zhang, H., Xu, H., Xiao, Y., Guo, X., & Ma, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6975 · Cited by 589 · cited 1×
“The commonly used method for discretization is the zero-order hold (ZOH) rule, which is defined as follows: After the discretization, the discretized version of Eq.”
FusionDN: A Unified Densely Connected Network for Image Fusion
Xu, H., Ma, J., Le, Z., Jiang, J., & Guo, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6936 · Cited by 587 · cited 7×
“FusionGAN’s fusion images have lower contrast and missing texture details, while CSMCA [13] and IFT [27] preserve texture information well.”
InfluentialDATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer
Tang, W., He, F., Liu, Y., Duan, Y., & Si, T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3234340 · Cited by 571
DIVFusion: Darkness-free infrared and visible image fusion
Tang, L., Xiang, X., Zhang, H., Gong, M., & Ma, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.10.034 · Cited by 509
YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer
Tang, W., He, F., & Liu, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3192661 · Cited by 485
The TNO Multiband Image Data Collection
Toet, A. (2017) · Data in Brief · doi:10.1016/j.dib.2017.09.038 · Cited by 476 · cited 2×
“6, FusionGAN and U2Fusion [30] exhibit unsatisfactory brightness and clarity, with some gray matter blurring the texture details.”
MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer
Tang, W., He, F., Liu, Y., & Duan, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3193288 · Cited by 442 · cited 7×
“U2Fusion [30] and MATR [24] can well retain texture details, but there is slight color distortion.”
InfluentialLi, J., Huo, H., Li, C., Wang, R., & Feng, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2997127 · Cited by 415
Visible and Infrared Image Fusion Using Deep Learning
Zhang, X., & Demiris, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261282 · Cited by 392 · cited 2×
“].”
Medical Image Fusion via Convolutional Sparsity Based Morphological Component Analysis
Liu, Y., Chen, X., Ward, R. K., & Wang, Z. J. (2019) · IEEE Signal Processing Letters · doi:10.1109/lsp.2019.2895749 · Cited by 315
A human perception inspired quality metric for image fusion based on regional information
Chen, H., & Varshney, P. K. (2007) · Information Fusion · doi:10.1016/j.inffus.2005.10.001 · Cited by 286 · cited 1×
“…as Fusion performance is evaluated using six key metrics [2, 19]: Structural Fidelity (SF), Structure Content Difference (SCD), Multiscale Structural Similarity Index Measure (MS-SSIM), Gradient-Based Metric (Q / ), Feature Mutual Information (FMI), and Visual Information Fidelity (VIF) [1].”
Liu, J., Lin, R., Wu, G., Liu, R., Luo, Z., & Fan, X. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01952-1 · Cited by 244 · cited 1×
“(3) can also be mathematically equivalently transformed into the following CNN form: where ⊛ denotes convolution operation, K ∈ R is a structured convolution kernel and L denotes the length of the input sequence x .”
Koroleva, O. A., Tomlinson, M. L., Leader, D., Shaw, P., & Doonan, J. H. (2005) · The Plant Journal · doi:10.1111/j.1365-313x.2004.02281.x · Cited by 219 · cited 3×
“…learning, the primary image fusion methodologies can be categorized into four groups: Convolutional Neural Network (CNN) models [30, 31, 37], Generative Ad-versarial Network (GAN) models [16, 17], Autoencoder (AE) based models [10–12, 20] , and models based on Transformer architectures [ 15, 24”
Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World
Wang, J., Liu, A., Yin, Z., Liu, S., Tang, S., & Liu, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00846 · Cited by 203 · cited 1×
“GFP images provide functional information related to protein distribution, while PC images contain rich details of cell structure including nucleus and mitochondria.”
Proceedings 2002 International Conference On Image Processing
(2002) · Proceedings International Conference on Image Processing ICIP-02 · doi:10.1109/icip.2002.1037942 · Cited by 180
Deep learning methods for medical image fusion: A review
Zhou, T., Cheng, Q., Lu, H., Li, Q., Zhang, X., & Qiu, S. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106959 · Cited by 170 · cited 1×
“Medical imaging’s MRI and CT provide structural insights [39], while PET offers functional data [ ?”
Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
Yu, Z., Zhou, B., Wan, J., Wang, P., Chen, H., Liu, X., Li, S. Z., & Zhao, G. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3087348 · Cited by 156 · cited 1×
“Fusion addresses sensitivity and noise issues, supporting practical applications [32–34].”
Self-citationA Similarity Metric For Assessment Of Image Fusion Algorithms
Cvejic, N., Łoza, A., Bull, D., & Canagarajah, N. C. (2008) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1329564 · Cited by 156
Multigrained Attention Network for Infrared and Visible Image Fusion
Li, J., Huo, H., Li, C., Wang, R.-H., Sui, C., & Liu, Z. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3029360 · Cited by 126 · cited 1×
“Mamba’s versatility and efficiency make it a compelling choice for a wide range of applications, from language processing to computer vision representation tasks [6, 18].”
MCFNet: Multi-Layer Concatenation Fusion Network for Medical Images Fusion
Liang, X., Hu, P., Zhang, L., Sun, J., & Yin, G. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2913281 · Cited by 121 · cited 1×
“Specifically, denote Δ as the timescale parameter to transform the continuous parameters A , B to discrete parameters A , B .”
Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing
Yu, Z., Cai, R., Cui, Y., Liu, X., Hu, Y., & Kot, A. C. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02055-1 · Cited by 103
Self-citationAyache, N., Sebastien Ourselin, A., & Maeder, A. J. (2007) · Cited by 99
AFT: Adaptive Fusion Transformer for Visible and Infrared Images
Chang, Z., Feng, Z., Yang, S., & Gao, Q. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3263113 · Cited by 91 · cited 1×
“Mamba’s versatility and efficiency make it a compelling choice for a wide range of applications, from language processing to computer vision representation tasks [6, 18].”
MACTFusion: Lightweight Cross Transformer for Adaptive Multimodal Medical Image Fusion
Xie, X., Zhang, X., Tang, X., Zhao, J., Xiong, D., Ouyang, L., Yang, B. F., Zhou, H., Ling, B. W., & Teo, K. L. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3391620 · Cited by 90
Self-citationCAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q., Yu, Z., Shao, R., Xie, X., Torr, P. H. S., & Cao, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_9 · Cited by 86
Self-citationXie, J., Lin, Y., Yan, L., & Chen, B. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3328080 · Cited by 50
Improving Skip Connection in U-Net Through Fusion Perspective With Mamba for Image Dehazing
Ju, M., Xie, S., & Li, F. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3417476 · Cited by 32
Zhang, T., Tan, T., Han, L., Wang, X., Gao, Y., van Dijk, J. M., Portaluri, A., González-Huete, A., D'Angelo, A. R., Lu, C., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102381 · Cited by 23 · cited 2×
“Therefore, in this section, we focus on introducing the designed dynamic vision state space module, dynamic feature enhancement module, and cross modality fusion Mamba module.”
Self-citation
Metadata
- Authors
- Xie, X., Cui, Y., Tan, T., Zheng, X., & Yu, Z.
- Venue
- Visual Intelligence
- Published
- 2024-12-31
- Volume
- 2
- Issue
- 1
- ISSN
- 2731-9008 (ISSN-L)
- OpenAlex
- W4405953164
- Cited by
- 249
- References
- 54
- FWCI
- 41.96
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.