The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification
Open accessAbstract
Classification and identification of the materials lying over or beneath the earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS), and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML)-cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on “what,” “where,” and “how” to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS data sets. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
(2011) · Cited by 2,752
Deep Learning for Hyperspectral Image Classification: An Overview
Li, S., Song, W., Fang, L., Chen, Y., Ghamisi, P., & Benediktsson, J. A. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2907932 · Cited by 2,007
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
Airborne laser scanning—an introduction and overview
Wehr, A., & Lohr, U. (1999) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/s0924-2716(99)00011-8 · Cited by 1,414
Advances in Spectral-Spatial Classification of Hyperspectral Images
Fauvel, M., Tarabalka, Y., Benediktsson, J. A., Chanussot, J., & Tilton, J. C. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2197589 · Cited by 1,282
Self-citationAn Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
Hong, D., Yokoya, N., Chanussot, J., & Zhu, X. X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2878958 · Cited by 948
Self-citation
Cited by
SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers
Hong, D., Han, Z., Yao, J., Gao, L., Zhang, B., Plaza, A., & Chanussot, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3130716 · Cited by 1,382
Self-citationUIU-Net: U-Net in U-Net for Infrared Small Object Detection
Wu, X., Hong, D., & Chanussot, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3228497 · Cited by 1,073
Self-citationDeep learning in multimodal remote sensing data fusion: A comprehensive review
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102926 · Cited by 630
Self-citationHong, D., Zhang, B., Li, H., Li, Y., Yao, J., Li, C., Werner, M., Chanussot, J., Zipf, A., & Zhu, X. X. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113856 · Cited by 545
Self-citationAttention-Based Adaptive Spectral–Spatial Kernel ResNet for Hyperspectral Image Classification
Roy, S. K., Manna, S., Song, T., & Bruzzone, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3043267 · Cited by 519
Multimodal Fusion Transformer for Remote Sensing Image Classification
Roy, S. K., Deria, A., Hong, D., Rasti, B., Plaza, A., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3286826 · Cited by 519
Self-citationHyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects
Ahmad, M., Shabbir, S., Roy, S. K., Hong, D., Wu, X., Yao, J., Khan, A. M., Mazzara, M., Distefano, S., & Chanussot, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3133021 · Cited by 438
Self-citationDeep learning for unmanned aerial vehicle-based object detection and tracking: a survey
Wu, X., Li, W., Hong, D., Tao, R., & Du, Q. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2021.3115137 · Cited by 421
Self-citationConvolutional Neural Networks for Multimodal Remote Sensing Data Classification
Wu, X., Hong, D., & Chanussot, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3124913 · Cited by 412
Self-citationYao, J., Zhang, B., Li, C., Hong, D., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3284671 · Cited by 390
Self-citationSuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery
Zhang, J., Lei, J., Xie, W., Fang, Z., Li, Y., & Du, Q. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3258666 · Cited by 388
Self-citationSpectral–Spatial Morphological Attention Transformer for Hyperspectral Image Classification
Roy, S. K., Deria, A., Shah, C., Haut, J. M., Du, Q., & Plaza, A. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3242346 · Cited by 371
Self-citationDeep learning for change detection in remote sensing: a review
Bai, T., Wang, L., Yin, D., Sun, K., Chen, Y., Li, W., & Li, D. (2023) · Geo-spatial Information Science · doi:10.1080/10095020.2022.2085633 · Cited by 328
Hong, D., Hu, J., Yao, J., Chanussot, J., & Zhu, X. X. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.05.011 · Cited by 326
Self-citationHyperspectral Image Transformer Classification Networks
Yang, X., Cao, W., Lu, Y., & Zhou, Y. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3171551 · Cited by 282
A comparative review on multi-modal sensors fusion based on deep learning
Tang, Q., Liang, J., & Zhu, F. (2023) · Signal Processing · doi:10.1016/j.sigpro.2023.109165 · Cited by 280
Hong, D., He, W., Yokoya, N., Yao, J., Gao, L., Zhang, L., Chanussot, J., & Zhu, X. (2021) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2021.3064051 · Cited by 275
Self-citationA Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation
Ma, X., Zhang, X., Pun, M.-O., & Liu, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3373033 · Cited by 265
Asymmetric Siamese Networks for Semantic Change Detection in Aerial Images
Yang, K., Xia, G.-S., Liu, Z., Du, B., Yang, W., Pelillo, M., & Zhang, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3113912 · Cited by 261
Big Data in Earth system science and progress towards a digital twin
Li, X., Feng, M., Ran, Y., Su, Y., Liu, F., Huang, C., Shen, H., Xiao, Q., Su, J., Yuan, S., et al. (2023) · Nature Reviews Earth & Environment · doi:10.1038/s43017-023-00409-w · Cited by 257
On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances, and Million-AID
Long, Y., Xia, G.-S., Li, S., Yang, W., Yang, M. Y., Zhu, X. X., Zhang, L., & Li, D. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3070368 · Cited by 250
Liang, C., Wang, L., Liu, L., Zhang, H., & Guo, F. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109632 · Cited by 72
MSMatch: Semisupervised Multispectral Scene Classification With Few Labels
Gomez, P., & Meoni, G. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3126082 · Cited by 55
Passos, L. A., Papa, J. P., Del Ser, J., Hussain, A., & Adeel, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.006 · Cited by 47
Shamsolmoali, P., Chanussot, J., Zareapoor, M., Zhou, H., & Yang, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3106442 · Cited by 44
“However, by analyzing more dimensions and adopting fusion methods [20], [21], we can train a network to achieve better performance and efficiency.”
MethodSelf-citation
Metadata
- Authors
- Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., & Zhang, B.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2021-05
- Volume
- 59
- Issue
- 5
- Pages
- 4340-4354
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W3048631361
- Cited by
- 1,374
- References
- 42
- FWCI
- 103.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.