The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification
Open accessAbstract
Classification and identification of the materials lying over or beneath the earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS), and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML)-cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on “what,” “where,” and “how” to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS data sets. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
(2011) · Cited by 2,752
Deep Learning for Hyperspectral Image Classification: An Overview
Li, S., Song, W., Fang, L., Chen, Y., Ghamisi, P., & Benediktsson, J. A. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2907932 · Cited by 2,007
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
Airborne laser scanning—an introduction and overview
Wehr, A., & Lohr, U. (1999) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/s0924-2716(99)00011-8 · Cited by 1,414
Advances in Spectral-Spatial Classification of Hyperspectral Images
Fauvel, M., Tarabalka, Y., Benediktsson, J. A., Chanussot, J., & Tilton, J. C. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2197589 · Cited by 1,282
Self-citationAn Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
Hong, D., Yokoya, N., Chanussot, J., & Zhu, X. X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2878958 · Cited by 948
Self-citationMultisource Remote Sensing Data Classification Based on Convolutional Neural Network
Xu, X., Li, W., Ran, Q., Du, Q., Gao, L., & Zhang, B. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2017.2756851 · Cited by 689
Self-citationBeyond RGB: Very high resolution urban remote sensing with multimodal deep networks
Audebert, N., Le Saux, B., & Lefèvre, S. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2017.11.011 · Cited by 655
Rasti, B., Hong, D., Hang, R., Ghamisi, P., Kang, X., Chanussot, J., & Benediktsson, J. A. (2020) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2020.2979764 · Cited by 537
Self-citationCascaded Recurrent Neural Networks for Hyperspectral Image Classification
Hang, R., Liu, Q., Hong, D., & Ghamisi, P. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2899129 · Cited by 534
Self-citationMultimodal Classification of Remote Sensing Images: A Review and Future Directions
Gomez-Chova, L., Tuia, D., Moser, G., & Camps-Valls, G. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2449668 · Cited by 508
CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM
Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., & Davison, A. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00271 · Cited by 372
Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks
Audebert, N., Le Saux, B., & Lefèvre, S. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_12 · Cited by 367
Hyperspectral Image Classification With Convolutional Neural Network and Active Learning
Cao, X., Yao, J., Xu, Z., & Meng, D. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2964627 · Cited by 365
Self-citationXu, Y., Du, B., Zhang, L., Cerra, D., Pato, M., Carmona, E., Prasad, S., Yokoya, N., Hansch, R., & Le Saux, B. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2911113 · Cited by 337
Self-citationHong, D., Wu, X., Ghamisi, P., Chanussot, J., Yokoya, N., & Zhu, X. X. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2957251 · Cited by 310
Self-citationX-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing data
Hong, D., Yokoya, N., Xia, G.-S., Chanussot, J., & Zhu, X. X. (2020) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2020.06.014 · Cited by 257
Self-citationHong, D., Yokoya, N., Ge, N., Chanussot, J., & Zhu, X. X. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2018.10.006 · Cited by 256
Self-citationHyperspectral and LiDAR Data Fusion Using Extinction Profiles and Deep Convolutional Neural Network
Ghamisi, P., Hofle, B., & Zhu, X. X. (2017) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2016.2634863 · Cited by 250
Deep Fusion of Remote Sensing Data for Accurate Classification
Chen, Y., Li, C., Ghamisi, P., Jia, X., & Gu, Y. (2017) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2017.2704625 · Cited by 245
Multi-source remotely sensed data fusion for improving land cover classification
Chen, B., Huang, B., & Xu, B. (2017) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2016.12.008 · Cited by 238
CoSpace: Common Subspace Learning From Hyperspectral-Multispectral Correspondences
Hong, D., Yokoya, N., Chanussot, J., & Zhu, X. X. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2890705 · Cited by 221
Self-citationStfNet: A Two-Stream Convolutional Neural Network for Spatiotemporal Image Fusion
Liu, X., Deng, C., Chanussot, J., Hong, D., & Zhao, B. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2907310 · Cited by 220
Self-citationNonconvex-Sparsity and Nonlocal-Smoothness-Based Blind Hyperspectral Unmixing
Yao, J., Meng, D., Zhao, Q., Cao, W., & Xu, Z. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2893068 · Cited by 164
Self-citationVolpi, M., Camps-Valls, G., & Tuia, D. (2015) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2015.02.005 · Cited by 158
Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component Analysis
Rasti, B., Ghamisi, P., Plaza, J., & Plaza, A. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2017.2726901 · Cited by 151
Yokoya, N., Ghamisi, P., Xia, J., Sukhanov, S., Heremans, R., Tankoyeu, I., Bechtel, B., Saux, B. L., Moser, G., & Tuia, D. (2018) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2018.2799698 · Cited by 151
Self-citationSrivastava, S., Vargas-Muñoz, J. E., & Tuia, D. (2019) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.04.014 · Cited by 151
Hong, D., Yokoya, N., Chanussot, J., Xu, J., & Zhu, X. X. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.09.008 · Cited by 143
Self-citationBenedetti, P., Ienco, D., Gaetano, R., Osé, K., Pensa, R. G., & Dupuy, S. (2018) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2018.2876357 · Cited by 139
Semisupervised Manifold Alignment of Multimodal Remote Sensing Images
Tuia, D., Volpi, M., Trolliet, M., & Camps‐Valls, G. (2014) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2014.2317499 · Cited by 137
Wu, X., Hong, D., Chanussot, J., Xu, Y., Tao, R., & Wang, Y. (2019) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2019.2919755 · Cited by 137
Self-citationA Novel MKL Model of Integrating LiDAR Data and MSI for Urban Area Classification
Gu, Y., Wang, Q., Jia, X., et al. (2015) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2015.2421051 · Cited by 122
Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images
Volpi, M., & Tuia, D. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2018.06.007 · Cited by 96
Hu, J., Hong, D., & Zhu, X. X. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2924113 · Cited by 78
Self-citationLearning-Shared Cross-Modality Representation Using Multispectral-LiDAR and Hyperspectral Data
Hong, D., Chanussot, J., Yokoya, N., Kang, J. S., & Zhu, X. X. (2020) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2019.2944599 · Cited by 64
Self-citationMayer, R. R., Bucholtz, F., & Scribner, D. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2003.813553 · Cited by 63
Malec, S., Rogge, D. M., Heiden, U., Sánchez-Azofeifa, A. G., Bachmann, M., & Wegmann, M. (2015) · Remote Sensing · doi:10.3390/rs70911776 · Cited by 18
Cited by
SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers
Hong, D., Han, Z., Yao, J., Gao, L., Zhang, B., Plaza, A., & Chanussot, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3130716 · Cited by 1,382
Self-citationUIU-Net: U-Net in U-Net for Infrared Small Object Detection
Wu, X., Hong, D., & Chanussot, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3228497 · Cited by 1,073
Self-citationDeep learning in multimodal remote sensing data fusion: A comprehensive review
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102926 · Cited by 630
Self-citationHong, D., Zhang, B., Li, H., Li, Y., Yao, J., Li, C., Werner, M., Chanussot, J., Zipf, A., & Zhu, X. X. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113856 · Cited by 545
Self-citationAttention-Based Adaptive Spectral–Spatial Kernel ResNet for Hyperspectral Image Classification
Roy, S. K., Manna, S., Song, T., & Bruzzone, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.3043267 · Cited by 519
Metadata
- Authors
- Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., & Zhang, B.
- Venue
- IEEE Transactions on Geoscience and Remote Sensing
- Published
- 2021-05
- Volume
- 59
- Issue
- 5
- Pages
- 4340-4354
- ISSN
- 0196-2892 (ISSN-L)
- 1558-0644
- OpenAlex
- W3048631361
- Cited by
- 1,374
- References
- 42
- FWCI
- 103.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.