Multimodal Fusion Transformer for Remote Sensing Image Classification
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a distinctive representation in a reduced and hierarchical feature space. Extensive experiments are carried out on widely used benchmark datasets i.e., the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at https://github.com/AnkurDeria/MFT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 1×
“As such, the order of spectral bands is essential in learning long-term dependencies and preventing gradient vanishing problems [47], which might further complicate the interpretation of spectrally salient changes.”
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Classification of hyperspectral remote sensing images with support vector machines
Melgani, F., & Bruzzone, L. (2004) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2004.831865 · Cited by 4,411
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · cited 1×
“analyzing sequential (or time series) data [48].”
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Zhong, Z., Li, J., Luo, Z., & Chapman, M. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2017.2755542 · Cited by 1,959
HybridSN: Exploring 3-D–2-D CNN Feature Hierarchy for Hyperspectral Image Classification
Roy, S. K., Krishna, G., Dubey, S. R., & Chaudhuri, B. B. (2020) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2019.2918719 · Cited by 1,898
Self-citationGLC2000: a new approach to global land cover mapping from Earth observation data
Bartholomé, E., & Belward, A. S. (2005) · International Journal of Remote Sensing · doi:10.1080/01431160412331291297 · Cited by 1,798
Graph Convolutional Networks for Hyperspectral Image Classification
Hong, D., Gao, L., Yao, J., Zhang, B., Plaza, A., & Chanussot, J. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3015157 · Cited by 1,691
Self-citationSpectralFormer: Rethinking Hyperspectral Image Classification With Transformers
Hong, D., Han, Z., Yao, J., Gao, L., Zhang, B., Plaza, A., & Chanussot, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3130716 · Cited by 1,382
Self-citationMore Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification
Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., & Zhang, B. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3016820 · Cited by 1,374
Self-citationUIU-Net: U-Net in U-Net for Infrared Small Object Detection
Wu, X., Hong, D., & Chanussot, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3228497 · Cited by 1,073
Self-citationDeep supervised learning for hyperspectral data classification through convolutional neural networks
Makantasis, K., Karantzalos, K., Doulamis, A., & Doulamis, N. (2015) · 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2015.7326945 · Cited by 1,011
An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
Hong, D., Yokoya, N., Chanussot, J., & Zhu, X. X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2878958 · Cited by 948
Self-citation3-D Deep Learning Approach for Remote Sensing Image Classification
Ben Hamida, A., Benoit, A., Lambert, P., & Ben Amar, C. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2818945 · Cited by 875
Spectral–Spatial Feature Tokenization Transformer for Hyperspectral Image Classification
Sun, L., Zhao, G., Zheng, Y., & Wu, Z. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3144158 · Cited by 778 · cited 1×
“Inspired by the success of transformers in computer vision [49], several new transformer models have been developed in the last few years [50].”
MethodGenerative Adversarial Networks for Hyperspectral Image Classification
Zhu, L., Chen, Y., Ghamisi, P., & Benediktsson, J. A. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2805286 · Cited by 734
Morphological Attribute Profiles for the Analysis of Very High Resolution Images
Dalla Mura, M., Benediktsson, J. A., Waske, B., & Bruzzone, L. (2010) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2010.2048116 · Cited by 722
Ghamisi, P., Gloaguen, R., Atkinson, P. M., Benediktsson, J. A., Rasti, B., Yokoya, N., Wang, Q., Hofle, B., Bruzzone, L., Bovolo, F., et al. (2019) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2018.2890023 · Cited by 629
Self-citationRasti, B., Hong, D., Hang, R., Ghamisi, P., Kang, X., Chanussot, J., & Benediktsson, J. A. (2020) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2020.2979764 · Cited by 537
Self-citationDeep Pyramidal Residual Networks for Spectral–Spatial Hyperspectral Image Classification
Paoletti, M. E., Haut, J. M., Fernandez-Beltran, R., Plaza, J., Plaza, A. J., & Pla, F. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2860125 · Cited by 531
Self-citationAttention-Based Adaptive Spectral–Spatial Kernel ResNet for Hyperspectral Image Classification
Roy, S. K., Manna, S., Song, T., & Bruzzone, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3043267 · Cited by 519
Self-citationResidual Spectral–Spatial Attention Network for Hyperspectral Image Classification
Zhu, M., Jiao, L., Liu, F., Yang, S., & Wang, J. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2994057 · Cited by 498
Classification of Hyperspectral and LiDAR Data Using Coupled CNNs
Hang, R., Li, Z., Ghamisi, P., Hong, D., Xia, G., & Liu, Q. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2969024 · Cited by 466
Self-citationHyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects
Ahmad, M., Shabbir, S., Roy, S. K., Hong, D., Wu, X., Yao, J., Khan, A. M., Mazzara, M., Distefano, S., & Chanussot, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3133021 · Cited by 438
Self-citationTransformers in Remote Sensing: A Survey
Aleissaee, A. A., Kumar, A., Anwer, R. M., Khan, S., Cholakkal, H., Xia, G.-S., & Khan, F. S. (2023) · Remote Sensing · doi:10.3390/rs15071860 · Cited by 417
Spectral–Spatial Morphological Attention Transformer for Hyperspectral Image Classification
Roy, S. K., Deria, A., Shah, C., Haut, J. M., Du, Q., & Plaza, A. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3242346 · Cited by 371
Self-citationNoise Reduction in Hyperspectral Imagery: Overview and Application
Rasti, B., Scheunders, P., Ghamisi, P., Licciardi, G., & Chanussot, J. (2018) · Remote Sensing · doi:10.3390/rs10030482 · Cited by 346
Self-citationHong, D., Hu, J., Yao, J., Chanussot, J., & Zhu, X. X. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.05.011 · Cited by 326
Self-citationHong, D., He, W., Yokoya, N., Yao, J., Gao, L., Zhang, L., Chanussot, J., & Zhu, X. (2021) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2021.3064051 · Cited by 275
Self-citationdiffGrad: An Optimization Method for Convolutional Neural Networks
Dubey, S. R., Chakraborty, S., Roy, S. K., Mukherjee, S., Singh, S. K., & Chaudhuri, B. B. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2955777 · Cited by 270
Self-citationFusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning
Khodadadzadeh, M., Li, J., Prasad, S., & Plaza, A. (2015) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2432037 · Cited by 211
Self-citationJoint Classification of Hyperspectral and LiDAR Data Using a Hierarchical CNN and Transformer
Zhao, G., Ye, Q., Sun, L., Wu, Z., Pan, C., & Jeon, B. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3232498 · Cited by 203
Koetz, B., Morsdorf, F., van der Linden, S., Curt, T., & Allgöwer, B. (2008) · Forest Ecology and Management · doi:10.1016/j.foreco.2008.04.025 · Cited by 186
Global–Local Transformer Network for HSI and LiDAR Data Joint Classification
Ding, K., Lu, T., Fu, W., Li, S., & Ma, F. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3216319 · Cited by 177
Wang, X., Feng, Y., Song, R., Mu, Z., & Song, C. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.008 · Cited by 164
Wu, X., Hong, D., Chanussot, J., Xu, Y., Tao, R., & Wang, Y. (2019) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2019.2919755 · Cited by 137
Self-citationMultimodal GANs: Toward Crossmodal Hyperspectral–Multispectral Image Segmentation
Hong, D., Yao, J. T., Meng, D., Xu, Z., & Chanussot, J. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3020823 · Cited by 131
Self-citationMorphological Convolutional Neural Networks for Hyperspectral Image Classification
Roy, S. K., Mondal, R., Paoletti, M. E., Haut, J. M., & Plaza, A. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3088228 · Cited by 117
Self-citationGenerative Adversarial Minority Oversampling for Spectral–Spatial Hyperspectral Image Classification
Roy, S. K., Haut, J. M., Paoletti, M. E., Dubey, S. R., & Plaza, A. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3052048 · Cited by 114
Self-citationHong, D., Yokoya, N., Chanussot, J., Xu, J., & Zhu, X. X. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3028931 · Cited by 108
Self-citationRoy, S. K., Chatterjee, S., Bhattacharyya, S., Chaudhuri, B. B., & Platoš, J. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2961681 · Cited by 102
Self-citationRoy, S. K., Dubey, S. R., Chatterjee, S., & Chaudhuri, B. B. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.1462 · Cited by 87
Self-citationLocal Information Interaction Transformer for Hyperspectral and LiDAR Data Classification
Zhang, Y., Peng, Y., Tu, B., & Liu, Y. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3232995 · Cited by 71
Chen, C., Yan, J., Wang, L., Liang, D., & Zhang, W. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3044250 · Cited by 58
Revisiting Deep Hyperspectral Feature Extraction Networks via Gradient Centralized Convolution
Roy, S. K., Kar, P., Hong, D., Wu, X., Plaza, A., & Chanussot, J. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3120198 · Cited by 53
Self-citationYu, Y., Jiang, T., Gao, J., Guan, H., Li, D., Gao, S., Tang, E., Wang, W., Tang, P., & Li, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102837 · Cited by 46
Rotation Equivariant Convolutional Neural Networks for Hyperspectral Image Classification
Paoletti, M. E., Haut, J. M., Roy, S. K., & Hendrix, E. M. T. (2020) · IEEE Access · doi:10.1109/access.2020.3027776 · Cited by 41
Self-citationAutomatic fusion and classification of hyperspectral and LiDAR data using random forests
Merentitis, A., Debes, C., Heremans, R., & Frangiadakis, N. (2014) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2014.6946658 · Cited by 24
A New Max-Min Convolutional Network for Hyperspectral Image Classification
Roy, S., Paoletti, M. E., Haut, J. M., Hendrix, E. M. T., & Plaza, A. (2021) · Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing · doi:10.1109/whispers52202.2021.9483983 · Cited by 11
Self-citation
Cited by
Yao, J., Zhang, B., Li, C., Hong, D., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3284671 · Cited by 390
Self-citationSpectral–Spatial Morphological Attention Transformer for Hyperspectral Image Classification
Roy, S. K., Deria, A., Shah, C., Haut, J. M., Du, Q., & Plaza, A. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3242346 · Cited by 371
Self-citationArtificial intelligence for geoscience: Progress, challenges, and perspectives
Zhao, T., Wang, S., Ouyang, C., Chen, M., Liu, C., Zhang, J., Yu, L., Wang, F., Xie, Y., Li, J., et al. (2024) · The Innovation · doi:10.1016/j.xinn.2024.100691 · Cited by 360
Self-citationA comparative review on multi-modal sensors fusion based on deep learning
Tang, Q., Liang, J., & Zhu, F. (2023) · Signal Processing · doi:10.1016/j.sigpro.2023.109165 · Cited by 280
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation
Ma, X., Zhang, X., Pun, M.-O., & Liu, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3373033 · Cited by 265
Metadata
- Authors
- Roy, S. K., Deria, A., Hong, D., Rasti, B., Plaza, A., & Chanussot, J.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2023
- Volume
- 61
- Pages
- 1-20
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W4380763457
- Cited by
- 519
- References
- 54
- FWCI
- 55.88
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.