CMTFNet: CNN and Multiscale Transformer Fusion Network for Remote-Sensing Image Semantic Segmentation
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Convolutional neural networks (CNNs) are powerful in extracting local information but lack the ability to model long-range dependencies. In contrast, transformer relies on multihead self-attention mechanisms to effectively extract the global contextual information and thus model long-range dependencies. In this paper, we propose a novel encoder-decoder structured semantic segmentation network, named as CNN and multiscale transformer fusion network (CMTFNet), to extract and fuse local information and multiscale global contextual information of high-resolution remote sensing images. Specifically, to further process the output features from the CNN encoder, we build a transformer decoder based on the multiscale multihead self-attention (M2SA) module for extracting rich multiscale global contextual information and channel information. Additionally, the transformer block introduces an efficient feed-forward network (E-FFN) to enhance the information interaction between different channels of the feature. Finally, the multiscale attention fusion (MAF) module fully fuses the feature information from different levels. We have conducted extensive comparison experiments and ablation experiments on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The extensive experimental results demonstrate that our proposed CMTFNet can obtain superior performance compared to the currently popular methods. The codes will be available at https://github.com/DrWuHonglin/CMTFNet.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
CBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517 · cited 1×
“[14], [15], [16], [17] proposed the DeepLab series of networks, which increases the receptive field without reducing the image size by applying the atrous convolution and proposed the atrous space pyramid pooling (ASPP) module for multiscale feature extraction.”
MethodSwin-Unet: Unet-like pure transformer for medical image segmentation
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25066-8_9 · Cited by 5,640
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1×
“Wang et al. [27] proposed the pyramid vision transformer (PVT) to reduce the computational overhead by using a spatial-reduction attention mechanism (SRA).”
BackgroundSegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213 · cited 1×
“Chen et al. [14], [15], [16], [17] proposed the DeepLab series of networks, which increases the receptive field without reducing the image size by applying the atrous convolution and proposed the atrous space pyramid pooling (ASPP) module for multiscale feature extraction.”
MethodRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1×
“segmentation tasks [22], [23], [24], [25].”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.15840 · Cited by 4,004
Random forest classifier for remote sensing classification
Pal, M. (2005) · International Journal of Remote Sensing · doi:10.1080/01431160412331269698 · Cited by 3,283
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,051
Unmanned Aerial Vehicle for Remote Sensing Applications—A Review
Yao, H., Qin, R., & Chen, X. (2019) · Remote Sensing · doi:10.3390/rs11121443 · Cited by 711
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation
He, X., Zhou, Y., Zhao, J., Zhang, D., Yao, R., & Xue, Y. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3144165 · Cited by 550
Li, R., Zheng, S., Zhang, C., Duan, C., Wang, L., & Atkinson, P. M. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.09.005 · Cited by 443
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images
Li, R., Zheng, S., Zhang, C., Duan, C., Su, J., Wang, L., & Atkinson, P. M. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3093977 · Cited by 341
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale AttentionPreprint
Wang, W., Lu, Y., Chen, L., Lin, B., Cai, D., He, X., & Liu, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00154 · Cited by 336
Single-Source Domain Expansion Network for Cross-Scene Hyperspectral Image Classification
Zhang, Y., Li, W., Sun, W., Tao, R., & Du, Q. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3243853 · Cited by 334
Zhang, C., Jiang, W., Zhang, Y., Wang, W., Zhao, Q., & Wang, C. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3144894 · Cited by 325
Multistage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
Li, R., Zheng, S., Duan, C., Su, J., & Zhang, C. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3063381 · Cited by 296
Li, H., Qiu, K., Chen, L., Mei, X., Hong, L., & Tao, C. (2021) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2020.2988294 · Cited by 295
Semantic labeling in very high resolution images via a self-cascaded convolutional neural network
Liu, Y., Fan, B., Wang, L., Bai, J., Xiang, S., & Pan, C. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.12.007 · Cited by 290
Wang, L., Li, R., Wang, D., Duan, C., Wang, T., & Meng, X. (2021) · Remote Sensing · doi:10.3390/rs13163065 · Cited by 281
Hyperspectral and LiDAR Data Classification Based on Structural Optimization Transmission
Zhang, M., Li, W., Zhang, Y., et al. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3169773 · Cited by 233
Self-citationEfficient Transformer for Remote Sensing Image Segmentation
Xu, Z., Zhang, W., Zhang, T., Yang, Z., & Li, J. (2021) · Remote Sensing · doi:10.3390/rs13183585 · Cited by 229
Hybrid Multiple Attention Network for Semantic Segmentation in Aerial Images
Niu, R., Sun, X., Tian, Y., Diao, W., Chen, K., & Fu, K. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3065112 · Cited by 221
A2-FPN for semantic segmentation of fine-resolution remotely sensed images
Li, R., Wang, L., Zhang, C., Duan, C., & Zheng, S. (2022) · International Journal of Remote Sensing · doi:10.1080/01431161.2022.2030071 · Cited by 213
Central Attention Network for Hyperspectral Imagery Classification
Liu, H., Li, W., Xia, X.-G., Zhang, M., Gao, C.-Z., & Tao, R. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3155114 · Cited by 181
Self-citationRefineNet: Multi-Path Refinement Networks for Dense Prediction
Lin, G., Liu, F., Milan, A., Shen, C., & Reid, I. D. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2893630 · Cited by 172
Road Extraction Methods in High-Resolution Remote Sensing Images: A Comprehensive Review
Lian, R., Wang, W., Mustafa, N., & Huang, L. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3023549 · Cited by 170
Li, R., Zheng, S., Duan, C., Wang, L., & Zhang, C. (2022) · Geo-spatial Information Science · doi:10.1080/10095020.2021.2017237 · Cited by 167
AFNet: Adaptive Fusion Network for Remote Sensing Image Semantic Segmentation
Liu, R., Li, M., & Chen, Z. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3034123 · Cited by 160
Guo, Y., Jia, X., & Paull, D. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2808767 · Cited by 139
Semantic Segmentation With Attention Mechanism for Remote Sensing Images
Zhao, Q., Liu, J., Li, Y., & Zhang, H. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3085889 · Cited by 136
Triplet adversarial domain adaptation for pixel-level classification of VHR remote sensing images
Yan, L., Fan, B., Liu, H., Huo, C., Xiang, S., & Pan, C. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2958123 · Cited by 115
Bai, L., Lin, X., Ye, Z., et al. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3185641 · Cited by 34
Cited by
RS 3 Mamba: Visual State Space Model for Remote Sensing Image Semantic Segmentation
Ma, X., Zhang, X., & Pun, M.-O. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3414293 · Cited by 376
RS-Mamba for Large Remote Sensing Image Dense Prediction
Zhao, S., Chen, H., Zhang, X., Xiao, P., Bai, L., & Ouyang, W. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3425540 · Cited by 323
Metadata
- Authors
- Wu, H., Huang, P., Zhang, M., Tang, W., & Yu, X.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2023
- Volume
- 61
- Pages
- 1-12
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W4386634500
- Cited by
- 360
- References
- 49
- FWCI
- 39.76
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.