The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
MHF-Net: An Interpretable Deep Network for Multispectral and Hyperspectral Image Fusion
Abstract
Multispectral and hyperspectral image fusion (MS/HS fusion) aims to fuse a high-resolution multispectral (HrMS) and a low-resolution hyperspectral (LrHS) images to generate a high-resolution hyperspectral (HrHS) image, which has become one of the most commonly addressed problems for hyperspectral image processing. In this paper, we specifically designed a network architecture for the MS/HS fusion task, called MHF-net, which not only contains clear interpretability, but also reasonably embeds the well studied linear mapping that links the HrHS image to HrMS and LrHS images. In particular, we first construct an MS/HS fusion model which merges the generalization models of low-resolution images and the low-rankness prior knowledge of HrHS image into a concise formulation, and then we build the proposed network by unfolding the proximal gradient algorithm for solving the proposed model. As a result of the careful design for the model and algorithm, all the fundamental modules in MHF-net have clear physical meanings and are thus easily interpretable. This not only greatly facilitates an easy intuitive observation and analysis on what happens inside the network, but also leads to its good generalization capability. Based on the architecture of MHF-net, we further design two deep learning regimes for two general cases in practice: consistent MHF-net and blind MHF-net. The former is suitable in the case that spectral and spatial responses of training and testing data are consistent, just as considered in most of the pervious general supervised MS/HS fusion researches. The latter ensures a good generalization in mismatch cases of spectral and spatial responses in training and testing data, and even across different sensors, which is generally considered to be a challenging issue for general supervised MS/HS fusion methods. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105
De-noising by soft-thresholding
Donoho, D. L. (1995) · IEEE Transactions on Information Theory · doi:10.1109/18.382009 · Cited by 9,619
FSIM: A Feature Similarity Index for Image Quality Assessment
Zhang, L., Zhang, L., Mou, X., & Zhang, D. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109730 · Cited by 5,203
Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches
Bioucas-Dias, J. M., Plaza, A., Dobigeon, N., Parente, M., Du, Q., Gader, P., & Chanussot, J. (2012) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2012.2194696 · Cited by 2,836
Measurement Reliability and Validity
(2000) · Research Methods in Applied Settings · doi:10.4324/9781410605337-29 · Cited by 2,541
ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing
Zhang, J., & Ghanem, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00196 · Cited by 1,309
Advances in Spectral-Spatial Classification of Hyperspectral Images
Fauvel, M., Tarabalka, Y., Benediktsson, J. A., Chanussot, J., & Tilton, J. C. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2197589 · Cited by 1,282
Improving Component Substitution Pansharpening Through Multivariate Regression of MS $+$Pan Data
Aiazzi, B., Baronti, S., & Selva, M. (2007) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2007.901007 · Cited by 1,223
A guide to convolution arithmetic for deep learning
Dumoulin, V., & Visin, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07285 · Cited by 1,172
Pansharpening by Convolutional Neural Networks
Masi, G., Cozzolino, D., Verdoliva, L., & Scarpa, G. (2016) · Remote Sensing · doi:10.3390/rs8070594 · Cited by 1,152
Detection algorithms for hyperspectral imaging applications
Manolakis, D., & Shaw, G. (2002) · IEEE Signal Processing Magazine · doi:10.1109/79.974724 · Cited by 1,116
Liu, J. G. (2000) · International Journal of Remote Sensing · doi:10.1080/014311600750037499 · Cited by 965
Hyperspectral Pansharpening: A Review
Loncan, L., de Almeida, L. B., Bioucas-Dias, J. M., Briottet, X., Chanussot, J., Dobigeon, N., Fabre, S., Liao, W., Licciardi, G. A., Simoes, M., et al. (2015) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2015.2440094 · Cited by 824
Hyperspectral and Multispectral Data Fusion: A comparative review of the recent literature
Yokoya, N., Grohnfeldt, C., & Chanussot, J. (2017) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2016.2637824 · Cited by 712
Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 GRSS Data Fusion Contest
Debes, C., Merentitis, A., Heremans, R., Hahn, J., Frangiadakis, N., van Kasteren, T., Liao, W., Bellens, R., Pizurica, A., Gautama, S., et al. (2014) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2014.2305441 · Cited by 653
Fusing Hyperspectral and Multispectral Images via Coupled Sparse Tensor Factorization
Li, S., Dian, R., Fang, L., & Bioucas-Dias, J. M. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2836307 · Cited by 585
The Laplacian Pyramid as a Compact Image Code
BURT, P. J., & ADELSON, E. H. (1987) · Readings in Computer Vision · doi:10.1016/b978-0-08-051581-6.50065-9 · Cited by 555
The Undecimated Wavelet Decomposition and its Reconstruction
Starck, J.-L., Fadili, J., & Murtagh, F. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.887733 · Cited by 530
Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions
Wald, L. (2002) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 520
Denoising Prior Driven Deep Neural Network for Image Restoration
Dong, W., Wang, P., Yin, W., Shi, G., Wu, F., & Lu, X. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2873610 · Cited by 515
Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network
Wei, Y., Yuan, Q., Shen, H., & Zhang, L. (2017) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2017.2736020 · Cited by 478
Fast Fusion of Multi-Band Images Based on Solving a Sylvester Equation
Wei, Q., Dobigeon, N., & Tourneret, J.-Y. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2458572 · Cited by 474
Hyperspectral Super-Resolution by Coupled Spectral Unmixing
Lanaras, C., Baltsavias, E., & Schindler, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.409 · Cited by 424
Target-Adaptive CNN-Based Pansharpening
Scarpa, G., Vitale, S., & Cozzolino, D. (2018) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2817393 · Cited by 383
A New Pansharpening Algorithm Based on Total Variation
Palsson, F., Sveinsson, J. R., & Ulfarsson, M. O. (2014) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2013.2257669 · Cited by 373
High-resolution hyperspectral imaging via matrix factorization
Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.-W., & Ikeuchi, K. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995457 · Cited by 370
Hyperspectral Image Classification With Convolutional Neural Network and Active Learning
Cao, X., Yao, J., Xu, Z., & Meng, D. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2964627 · Cited by 365
Self-citationMultispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network
Palsson, F., Sveinsson, J. R., & Ulfarsson, M. O. (2017) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2017.2668299 · Cited by 364
Remote Sensing Image Fusion With Deep Convolutional Neural Network
Shao, Z., & Cai, J. (2018) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2018.2805923 · Cited by 331
Kronecker-Basis-Representation Based Tensor Sparsity and Its Applications to Tensor Recovery
Xie, Q., Zhao, Q., Meng, D., & Xu, Z. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2734888 · Cited by 318
Self-citationProximal Dehaze-Net: A Prior Learning-Based Deep Network for Single Image Dehazing
Yang, D., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_43 · Cited by 315
Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net
Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., & Xu, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00168 · Cited by 302
Self-citationSparse Spatio-spectral Representation for Hyperspectral Image Super-resolution
Akhtar, N., Shafait, F., & Mian, A. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_5 · Cited by 284
Tarabalka, Y., Chanussot, J., & Benediktsson, J. A. (2010) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2009.2037132 · Cited by 274
Unsupervised Sparse Dirichlet-Net for Hyperspectral Image Super-Resolution
Qu, Y., Qi, H., & Kwan, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00266 · Cited by 259
Hyper-Sharpening: A First Approach on SIM-GA Data
Selva, M., Aiazzi, B., Butera, F., Chiarantini, L., & Baronti, S. (2015) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2440092 · Cited by 256
Pansharpening via Detail Injection Based Convolutional Neural Networks
He, L., Rao, Y., Li, J., Chanussot, J., Plaza, A., Zhu, J., & Li, B. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2898574 · Cited by 255
Noise-Resistant Wavelet-Based Bayesian Fusion of Multispectral and Hyperspectral Images
Zhang, Y., De Backer, S., & Scheunders, P. (2009) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2009.2017737 · Cited by 237
Spatial and Spectral Image Fusion Using Sparse Matrix Factorization
Huang, B., Song, H., Cui, H., Peng, J., & Xu, Z. (2014) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2013.2253612 · Cited by 231
Self-citationLearning Fully Convolutional Networks for Iterative Non-blind Deconvolution
Zhang, J., Pan, J., Lai, W.-S., Lau, R. W. H., & Yang, M.-H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.737 · Cited by 190
Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior
Wang, L., Sun, C., Fu, Y., Kim, M. H., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00822 · Cited by 186
Robust Online Matrix Factorization for Dynamic Background Subtraction
Yong, H., Meng, D., Zuo, W., & Lei Zhang, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2732350 · Cited by 182
Self-citationHyperPNN: Hyperspectral Pansharpening via Spectrally Predictive Convolutional Neural Networks
He, L., Zhu, J., Li, J., Plaza, A., Chanussot, J., & Li, B. H. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2917584 · Cited by 140
Hyperspectral imagery super-resolution by sparse representation and spectral regularization
Zhao, Y., Yang, J., Zhang, Q., Song, L., Cheng, Y., & Pan, Q. (2011) · EURASIP Journal on Advances in Signal Processing · doi:10.1186/1687-6180-2011-87 · Cited by 93
Hyperspectral Face Recognition using 3D-DCT and Partial Least Squares
Uzair, M., Mahmood, A., & Mian, A. S. (2013) · British Machine Vision Conference (BMVC) · doi:10.5244/c.27.57 · Cited by 89
Fusion of satellite images in urban area: Assessing the quality of resulting images
Zeng, Y., Huang, W., Liu, M., Zhang, H., & Zou, B. (2010) · International Conference on Geoinformatics · doi:10.1109/geoinformatics.2010.5568105 · Cited by 87
Fusion of Hyperspectral and Multispectral Images Using Spectral Unmixing and Sparse Coding
Nezhad, Z. H., Karami, A., Heylen, R., & Scheunders, P. (2016) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2016.2528339 · Cited by 85
A residual convolutional neural network for pan-shaprening
Rao, Y., He, L., & Zhu, J. (2017) · International Workshop on Remote Sensing with Intelligent Processing (RSIP) · doi:10.1109/rsip.2017.7958807 · Cited by 74
Molina, R., Vega, M., Mateos, J., & Katsaggelos, A. K. (2007) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2007.03.006 · Cited by 61
Zhang, Y., Wang, Y., Liu, Y., Zhang, C., He, M., & Mei, S. (2015) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss.2015.7326172 · Cited by 19
Cited by
Image fusion meets deep learning: A survey and perspective
Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.008 · Cited by 834
Deep learning in multimodal remote sensing data fusion: A comprehensive review
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102926 · Cited by 630
YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer
Tang, W., He, F., & Liu, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3192661 · Cited by 485
MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer
Tang, W., He, F., Liu, Y., & Duan, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3193288 · Cited by 442
Himeur, Y., Rimal, B., Tiwary, A., & Amira, A. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.06.003 · Cited by 288
Metadata
- Authors
- Xie, Q., Zhou, M., Zhao, Q., Xu, Z., & Meng, D.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-03-01
- Volume
- 44
- Issue
- 3
- Pages
- 1457-1473
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3048794210
- Cited by
- 268
- References
- 53
- FWCI
- 14.03
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.