The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation
Abstract
Accurate semantic segmentation of remote sensing data plays a crucial role in the success of geoscience research and applications. Recently, multimodal fusion-based segmentation models have attracted much attention due to their outstanding performance as compared to conventional single-modal techniques. However, most of these models perform their fusion operation using convolutional neural networks (CNN) or the vision transformer (Vit), resulting in insufficient local-global contextual modeling and representative capabilities. In this work, a multilevel multimodal fusion scheme called FTransUNet is proposed to provide a robust and effective multimodal fusion backbone for semantic segmentation by integrating both CNN and Vit into one unified fusion framework. Firstly, the shallow-level features are first extracted and fused through convolutional layers and shallow-level feature fusion (SFF) modules. After that, deep-level features characterizing semantic information and spatial relationships are extracted and fused by a well-designed Fusion Vit (FVit). It applies Adaptively Mutually Boosted Attention (Ada-MBA) layers and Self-Attention (SA) layers alternately in a three-stage scheme to learn cross-modality representations of high inter-class separability and low intra-class variations. Specifically, the proposed Ada-MBA computes SA and Cross-Attention (CA) in parallel to enhance intra- and cross-modality contextual information simultaneously while steering attention distribution towards semantic-aware regions. As a result, FTransUNet can fuse shallow-level and deep-level features in a multilevel manner, taking full advantage of CNN and transformer to accurately characterize local details and global semantics, respectively. Extensive experiments confirm the superior performance of the proposed FTransUNet compared with other multimodal fusion approaches on two fine-resolution remote sensing datasets, namely ISPRS Vaihingen and Potsdam. The source code in this work is available at https://github.com/sstary/SSRS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Self-citationU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
Swin-Unet: Unet-like pure transformer for medical image segmentation
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25066-8_9 · Cited by 5,640
Self-citationMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · cited 7×
“Empowered with its capability of sequence-to-sequence modeling, the Vit demonstrates significantly improved performance in extracting global contextual information as compared to the CNN-based methods, which has inspired many following works [31], [42], [43], [44], [45], [46].”
BackgroundMethodInfluentialEfficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 1×
“More recently, the transformer architecture has been considered for multimodal fusion in semantic segmentation for its outstanding capability in extracting contextual information [20], [26], [32], [54] and fusing different modalities [19], [38], [55], [56], [57], [58], [59].”
MethodRandom Forests for land cover classification
Gislason, P. O., Benediktsson, J. A., & Sveinsson, J. R. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.08.011 · Cited by 2,244
ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data
Diakogiannis, F. I., Waldner, F., Caccetta, P., & Wu, C. (2020) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2020.01.013 · Cited by 2,153
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,051
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification
Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., & Zhang, B. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3016820 · Cited by 1,374
Wang, L., Li, R., Zhang, C., Fang, S., Duan, C., Meng, X., & Atkinson, P. M. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.06.008 · Cited by 1,349
Wang, H., Cao, P., Wang, J., & Zaiane, O. R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20144 · Cited by 1,111
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., & Tai, C.-L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00116 · Cited by 990 · cited 1×
“However, only a few works were devoted to the multimodal fusion tasks, emphasizing the fusion of multimodal information according to the characteristics of different modalities [14], [33], [34].”
BackgroundFuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation
Lin, A., Chen, B., Xu, J., Zhang, Z., Lu, G., & Zhang, D. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3178991 · Cited by 882
Hyperspectral and Multispectral Image Fusion Based on a Sparse Representation
Wei, Q., Bioucas-Dias, J., Dobigeon, N., & Tourneret, J.-Y. (2015) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2014.2381272 · Cited by 698
Classification with an edge: Improving semantic image segmentation with boundary detection
Marmanis, D., Schindler, K., Wegner, J. D., Galliani, S., Datcu, M., & Stilla, U. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.11.009 · Cited by 670
Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks
Audebert, N., Le Saux, B., & Lefèvre, S. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.11.011 · Cited by 655
Deep learning in multimodal remote sensing data fusion: A comprehensive review
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102926 · Cited by 630
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Prakash, A., Chitta, K., & Geiger, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00700 · Cited by 584
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation
He, X., Zhou, Y., Zhao, J., Zhang, D., Yao, R., & Xue, Y. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3144165 · Cited by 550
Multimodal Fusion Transformer for Remote Sensing Image Classification
Roy, S. K., Deria, A., Hong, D., Rasti, B., Plaza, A., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3286826 · Cited by 519
Chen, X., Lin, K.-Y., Wang, J., Wu, W., Qian, C., Li, H., & Zeng, G. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_33 · Cited by 488
Li, R., Zheng, S., Zhang, C., Duan, C., Wang, L., & Atkinson, P. M. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.09.005 · Cited by 443
A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images
Wang, L., Li, R., Duan, C., Zhang, C., Meng, X., & Fang, S. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3143368 · Cited by 414
Convolutional Neural Networks for Multimodal Remote Sensing Data Classification
Wu, X., Hong, D., & Chanussot, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3124913 · Cited by 412
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis
Seichter, D., Kohler, M., Lewandowski, B., Wengefeld, T., & Gross, H.-M. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561675 · Cited by 330 · cited 2×
“…benchmarked the performance of the proposed FTransUNet against twelve representative state-of-the-art methods, namely ABCNet [61], PSPNet [62], MAResU-Net [63], vFuseNet [15], FuseNet [14], ESANet [33], Tran-sUNet [31], SA-GATE [64], UNetFormer [44], CMFNet [19], MFTransNet [59], and CMGFNet [47].”
BackgroundMethodHong, D., Hu, J., Yao, J., Chanussot, J., & Zhu, X. X. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.05.011 · Cited by 326
Multistage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
Li, R., Zheng, S., Duan, C., Su, J., & Zhang, C. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3063381 · Cited by 296
Deep Encoder–Decoder Networks for Classification of Hyperspectral and LiDAR Data
Hong, D., Gao, L., Hang, R., Zhang, B., & Chanussot, J. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2020.3017414 · Cited by 276
Gao, L., Liu, H., Yang, M., Chen, L., Wan, Y., Xiao, Z., & Qian, Y. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3119654 · Cited by 247
Hosseinpour, H., Samadzadegan, F., & Javan, F. D. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.12.007 · Cited by 227
Zhang, X., Yu, W., Pun, M.-O., & Shi, W. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.01.018 · Cited by 191
Self-citationModel-Based Fusion of Multi- and Hyperspectral Images Using PCA and Wavelets
Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., & Benediktsson, J. A. (2015) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2014.2363477 · Cited by 188
Touati, R., Mignotte, M., & Dahmane, M. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2933747 · Cited by 179
A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data
Ma, X., Zhang, X., & Pun, M.-O. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3165005 · Cited by 179
Self-citationCode-Aligned Autoencoders for Unsupervised Change Detection in Multimodal Remote Sensing Images
Luppino, L. T., Hansen, M. A., Kampffmeyer, M., Bianchi, F. M., Moser, G., Jenssen, R., & Anfinsen, S. N. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3172183 · Cited by 170
Dynamic Multicontext Segmentation of Remote Sensing Images Based on Convolutional Networks
Nogueira, K., Mura, M. D., Chanussot, J., Schwartz, W. R., & dos Santos, J. A. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2913861 · Cited by 164
Li, Y., Zhou, Y., Zhang, Y., Zhong, L., Wang, J., & Chen, J. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.02.013 · Cited by 162
Subspace-Based Support Vector Machines for Hyperspectral Image Classification
Gao, L., Li, J., Khodadadzadeh, M., Plaza, A., Zhang, B., He, Z., & Yan, H. (2014) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2014.2341044 · Cited by 141
Zhu, J., Guo, Y., Sun, G., Yang, L., Deng, M., & Chen, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3243042 · Cited by 134
Zhou, W., Jianhui, J., Lei, J., & Yu, L. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3159032 · Cited by 99
Multimodal Remote Sensing Image Segmentation With Intuition-Inspired Hypergraph Modeling
He, Q., Sun, X., Diao, W., Yan, Z., Yao, F., & Fu, K. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3245324 · Cited by 91
Ma, X., Zhang, X., Wang, Z., & Pun, M.-O. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3240982 · Cited by 75
Self-citationZhang, X., Zhang, B., Yu, W., & Kang, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3244136 · Cited by 63
Self-citationHe, S., Yang, H., Zhang, X., & Li, X. (2023) · Mathematics · doi:10.3390/math11030722 · Cited by 59
Self-citationZhang, X., Yu, W., & Pun, M.-O. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3157721 · Cited by 40
Self-citationXu, Z., Shen, Z., Li, Y., Xia, L., Wang, H., Li, S., Jiao, S., & Lei, Y. (2020) · Remote Sensing · doi:10.3390/rs13010090 · Cited by 34
Efficient Depth Fusion Transformer for Aerial Image Semantic Segmentation
Yan, L., Huang, J., Xie, H., Wei, P., & Gao, Z. (2022) · Remote Sensing · doi:10.3390/rs14051294 · Cited by 33
Synergetic Classification of Long-Wave Infrared Hyperspectral and Visible Images
Lu, X., Zhang, J., Li, T., & Zhang, G. (2015) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2442594 · Cited by 26
Adjacent Bi-Hierarchical Network for Scene Parsing of Remote Sensing Images
Ma, J., Zhou, W., Lei, J., & Yu, L. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3241648 · Cited by 21
Cited by
RS 3 Mamba: Visual State Space Model for Remote Sensing Image Semantic Segmentation
Ma, X., Zhang, X., & Pun, M.-O. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3414293 · Cited by 376
Self-citationA Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications
Jiao, T., Guo, C., Feng, X., Chen, Y., & Song, J. (2024) · Computers, Materials & Continua · doi:10.32604/cmc.2024.053204 · Cited by 158
Metadata
- Authors
- Ma, X., Zhang, X., Pun, M.-O., & Liu, M.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2024
- Volume
- 62
- Pages
- 1-15
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W4392543906
- Cited by
- 265
- References
- 60
- FWCI
- 44.66
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.