The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion
Abstract
Image fusion integrates a series of images acquired from different sensors,e.g., infrared and visible, outputting an image with richer information than either one. Traditional and recent deep-based methods have difficulties in preserving prominent structures and recovering vital textural details for practical applications. In this article, we propose a deep network for infrared and visible image fusion cascading a feature learning module with a fusion learning mechanism. Firstly, we apply a coarse-to-fine deep architecture to learn multi-scale features for multi-modal images, which enables discovering prominent common structures for later fusion operations. The proposed feature learning module requires no well-aligned image pairs for training. Compared with the existing learning-based methods, the proposed feature learning module can ensemble numerous examples from respective modals for training, increasing the ability of feature representation. Secondly, we design an edge-guided attention mechanism upon the multi-scale features to guide the fusion focusing on common structures, thus recovering details while attenuating noise. Moreover, we provide a new aligned infrared and visible image fusion dataset, RealStreet, collected in various practical scenarios for comprehensive evaluation. Extensive experiments on two benchmarks, TNO and RealStreet, demonstrate the superiority of the proposed method over the state-of-the-art in terms of both visual inspection and objective analysis on six evaluation metrics. We also conduct the experiments on the FLIR and NIR datasets, containing foggy weather and poor light conditions, to verify the generalization and robustness of the proposed method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
Dual Attention Network for Scene Segmentation
Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., & Lu, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00326 · Cited by 7,351
Self-citationImproved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800
Residual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073
FusionGAN: A generative adversarial network for infrared and visible image fusion
Ma, J., Yu, W., Liang, P., Li, C., & Jiang, J. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.09.004 · Cited by 2,267
Self-citationThe Nonsubsampled Contourlet Transform: Theory, Design, and Applications
Da Cunha, A. L., Zhou, J., & Do, M. N. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.877507 · Cited by 2,035
DenseFuse: A Fusion Approach to Infrared and Visible Images
Li, H., & Wu, X.-J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2887342 · Cited by 2,032
Objective image fusion performance measure
Xydeas, C. S., & Petrović, V. (2000) · Electronics Letters · doi:10.1049/el:20000267 · Cited by 1,996
Image Fusion With Guided Filtering
Li, S., Kang, X., & Hu, J. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2244222 · Cited by 1,781
Information measure for performance of image fusion
Qu, G., Zhang, D., & Yan, P. (2002) · Electronics Letters · doi:10.1049/el:20020212 · Cited by 1,427
Attention to Scale: Scale-Aware Semantic Image Segmentation
Chen, L.-C., Yang, Y., Wang, J., Xu, W., & Yuille, A. L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.396 · Cited by 1,377
Multi-focus image fusion with a deep convolutional neural network
Liu, Y., Chen, X., Peng, H., & Wang, Z. (2017) · Information Fusion · doi:10.1016/j.inffus.2016.12.001 · Cited by 1,327
EGNet: Edge Guidance Network for Salient Object Detection
Zhao, J., Liu, J.-J., Fan, D.-P., Cao, Y., Yang, J., & Cheng, M.-M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00887 · Cited by 1,202
Self-citationInfrared and visible image fusion via gradient transfer and total variation minimization
Ma, J., Chen, C., Li, C., & Huang, J. (2016) · Information Fusion · doi:10.1016/j.inffus.2016.02.001 · Cited by 1,128
A new image fusion performance metric based on visual information fidelity
Han, Y., Cai, Y., Cao, Y., & Xu, X. (2013) · Information Fusion · doi:10.1016/j.inffus.2011.08.002 · Cited by 1,071
Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism
Li, Y., Zeng, J., Shan, S., & Chen, X. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2886767 · Cited by 916
PiCANet: Learning Pixel-Wise Contextual Attention for Saliency Detection
Liu, N., Han, J., & Yang, M.-H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00326 · Cited by 863
Pixel- and region-based image fusion with complex wavelets
Lewis, J. J., O’Callaghan, R. J., Nikolov, S. G., Bull, D. R., & Canagarajah, N. (2007) · Information Fusion · doi:10.1016/j.inffus.2005.09.006 · Cited by 834
Ma, J., Zhou, Z., Wang, B., & Zong, H. (2017) · Infrared Physics & Technology · doi:10.1016/j.infrared.2017.02.005 · Cited by 773
Assessment of image fusion procedures using entropy, image quality, and multispectral classification
Van Aardt, J. (2008) · Journal of Applied Remote Sensing · doi:10.1117/1.2945910 · Cited by 747
Cui, G., Feng, H., Xu, Z., Li, Q., & Chen, Y. (2015) · Optics Communications · doi:10.1016/j.optcom.2014.12.032 · Cited by 612
Yin, M., Liu, X., Liu, Y., & Chen, X. (2019) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2018.2838778 · Cited by 594
Image fusion based on pixel significance using cross bilateral filter
Shreyamsha Kumar, B. K. (2015) · Signal, Image and Video Processing · doi:10.1007/s11760-013-0556-9 · Cited by 567
Infrared and Visible Image Fusion using a Deep Learning Framework
Li, H., Wu, X.-J., & Kittler, J. (2018) · 2018 24th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2018.8546006 · Cited by 534
A new image quality metric for image fusion: The sum of the correlations of differences
Aslantas, V., & Bendes, E. (2015) · AEU - International Journal of Electronics and Communications · doi:10.1016/j.aeue.2015.09.004 · Cited by 507
Zhang, Q., Liu, Y., Blum, R. S., Han, J., & Tao, D. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.05.006 · Cited by 499
Directive Contrast Based Multimodal Medical Image Fusion in NSCT Domain
Bhatnagar, G., Wu, Q. M. J., & Liu, Z. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2244870 · Cited by 471
Multi-spectral SIFT for scene category recognition
Brown, M., & Susstrunk, S. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995637 · Cited by 469
A novel multi-modality image fusion method based on image decomposition and sparse representation
Zhu, Z., Yin, H., Chai, Y., Li, Y., & Qi, G. (2018) · Information Sciences · doi:10.1016/j.ins.2017.09.010 · Cited by 408
STAT: Spatial-Temporal Attention Mechanism for Video Captioning
Yan, C., Tu, Y., Wang, X., Zhang, Y., Hao, X., Zhang, Y., & Dai, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2924576 · Cited by 399
Video Summarization With Attention-Based Encoder–Decoder Networks
Ji, Z., Xiong, K., Pang, Y., & Li, X. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2904996 · Cited by 392
DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion
Zhao, Z., Xu, S., Zhang, C., Liu, J., Zhang, J., & Li, P. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/135 · Cited by 341
Self-citationInfrared and visible image fusion method based on saliency detection in sparse domain
Liu, C. H., Qi, Y., & Ding, W. R. (2017) · Infrared Physics & Technology · doi:10.1016/j.infrared.2017.04.018 · Cited by 275
VIFB: A Visible and Infrared Image Fusion Benchmark
Zhang, X., Ye, P., & Xiao, G. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00060 · Cited by 250
ET-Net: A Generic Edge-aTtention Guidance Network for Medical Image Segmentation
Zhang, Z., Fu, H., Dai, H., Shen, J., Pang, Y., & Shao, L. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32239-7_49 · Cited by 246
Object fusion tracking based on visible and infrared images: A comprehensive review
Zhang, X., Ye, P., Leung, H., Gong, K., & Xiao, G. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.05.002 · Cited by 208
Joint patch clustering-based dictionary learning for multimodal image fusion
Kim, M., Han, D. K., & Ko, H. (2016) · Information Fusion · doi:10.1016/j.inffus.2015.03.003 · Cited by 196
Banach Wasserstein GANPreprint
Adler, J., & Lunz, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06621 · Cited by 189
IMAGE FUSION AND IMAGE QUALITY ASSESSMENT OF FUSED IMAGES
Han, Z., Tang, X., Gao, X., & Hu, F. (2013) · The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences · doi:10.5194/isprsarchives-xl-7-w1-33-2013 · Cited by 176
Zhao, W., Wang, D., & Lu, H. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2821177 · Cited by 168
A Bilevel Integrated Model With Data-Driven Layer Ensemble for Multi-Modality Image Fusion
Liu, R., Liu, J., Jiang, Z., Fan, X., & Luo, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3043125 · Cited by 165
Self-citationBlind Image Quality Assessment Based on Multichannel Feature Fusion and Label Transfer
Wu, Q., Li, H., Meng, F., Ngan, K. N., Luo, B., Huang, C., & Zeng, B. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2412773 · Cited by 164
Bayesian fusion for infrared and visible images
Zhao, Z., Xu, S., Zhang, C., Liu, J., & Zhang, J. (2020) · Signal Processing · doi:10.1016/j.sigpro.2020.107734 · Cited by 164
Self-citationRevisiting Feature Fusion for RGB-T Salient Object Detection
Zhang, Q., Xiao, T., Huang, N., Zhang, D., & Han, J. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3014663 · Cited by 164
Multisensor Image Fusion and Enhancement in Spectral Total Variation Domain
Zhao, W., Lu, H., & Wang, D. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2760100 · Cited by 148
Suganuma, M., Liu, X., & Okatani, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00925 · Cited by 121
Infrared and visible image fusion using fuzzy logic and population-based optimization
Saeedi, J., & Faez, K. (2011) · Applied Soft Computing · doi:10.1016/j.asoc.2011.11.020 · Cited by 116
Xu, H., Liang, P., Yu, W., Jiang, J., & Ma, J. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/549 · Cited by 112
Self-citationHu, H., Wu, J., Li, B. H., Guo, Q., & Zheng, J. Y. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2711422 · Cited by 88
On the Convergence of Learning-Based Iterative Methods for Nonconvex Inverse Problems
Liu, R., Cheng, S., He, Y., Fan, X., Lin, Z., & Luo, Z. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2920591 · Cited by 81
Self-citationContrast-Enhanced Fusion of Multisensor Images Using Subband-Decomposed Multiscale Retinex
Jang, J. H., Bae, Y., & Ra, J. B. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2197014 · Cited by 79
The infrared and visible image fusion algorithm based on target separation and sparse representation
Lu, X., Zhang, B., Zhao, Y., He, L., & Pei, H. (2014) · Infrared Physics & Technology · doi:10.1016/j.infrared.2014.09.007 · Cited by 79
Multi-Source Information Exchange Encoding With PCNN for Medical Image Fusion
Nie, R., Cao, J., Zhou, D., et al. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2998696 · Cited by 77
Knowledge-Driven Deep Unrolling for Robust Image Layer Separation
Liu, R., Jiang, Z., Fan, X., & Luo, Z. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2921597 · Cited by 67
Self-citationMulti-Scale Attention Deep Neural Network for Fast Accurate Object Detection
Song, K., Yang, H. S., & Yin, Z. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2875449 · Cited by 66
Kwak, J.-Y., Ko, B. C., & Nam, J. Y. (2016) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2016.2569159 · Cited by 61
Semantic Road Segmentation via Multi-scale Ensembles of Learned Features
Álvarez, J. M., LeCun, Y., Gevers, T., & López, A. M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33868-7_58 · Cited by 58
GraphNet: Learning Image Pseudo Annotations for Weakly-Supervised Semantic Segmentation
Pu, M., Huang, Y., Guan, Q., & Zou, Q. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240542 · Cited by 58
Low Dose Abdominal CT Image Reconstruction: An Unsupervised Learning Based Approach
Kuanar, S. P., Athitsos, V., Mahapatra, D., Rao, K. R., Akhtar, Z., & Dasgupta, D. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803037 · Cited by 56
Fusing Infrared and Visible Images of Different Resolutions via Total Variation Model
Du, Q., Xu, H., Ma, Y., et al. (2018) · Sensors · doi:10.3390/s18113827 · Cited by 53
Infrared and visible image fusion with spectral graph wavelet transform
Yan, X., Qin, H., Li, J., Zhou, H., & Zong, J. (2015) · Journal of the Optical Society of America A · doi:10.1364/josaa.32.001643 · Cited by 52
Deep Learning Based HEVC In-Loop Filtering for Decoder Quality Enhancement
Kuanar, S. P., Conly, C., & Rao, K. R. (2018) · Picture Coding Symposium · doi:10.1109/pcs.2018.8456278 · Cited by 50
Near-Infrared-Based Nighttime Pedestrian Detection Using Grouped Part Models
Lee, Y.-S., Chan, Y.-M., Fu, L., & Hsiao, P. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2385707 · Cited by 36
Infrared and visible image fusion based on total variation and augmented Lagrangian
Guo, H., Ma, Y., Mei, X., et al. (2017) · Journal of the Optical Society of America A · doi:10.1364/josaa.34.001961 · Cited by 33
Investigating Task-Driven Latent Feasibility for Nonconvex Image Modeling
Liu, R., Mu, P., Chen, J., Fan, X., & Luo, Z. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3004733 · Cited by 31
Self-citationSuperpixel-Based Causal Multisensor Video Fusion
Gangapure, V. N., Nanda, S., & Chowdhury, A. S. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2662743 · Cited by 28
Pedestrian Detection via Bi-directional Multi-scale Analysis
Duan, Z., Lan, J., Xu, Y., Ni, B., Zhuang, L., & Yang, X. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123356 · Cited by 13
Cited by
Liu, J., Fan, X., Huang, Z., Wu, G., Liu, R., Zhong, W., & Luo, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00571 · Cited by 1,336
Self-citationCDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Xu, S., Lin, Z., Timofte, R., & Van Gool, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00572 · Cited by 948
DATFuse: Infrared and Visible Image Fusion via Dual Attention Transformer
Tang, W., He, F., Liu, Y., Duan, Y., & Si, T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3234340 · Cited by 571
SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness
Tang, L., Deng, Y., Ma, Y., Huang, J., & Ma, J. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.106082 · Cited by 488
YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer
Tang, W., He, F., & Liu, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3192661 · Cited by 485
Metadata
- Authors
- Liu, J., Fan, X., Jiang, J., Liu, R., & Luo, Z.
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2022-01
- Volume
- 32
- Issue
- 1
- Pages
- 105-119
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W3126855404
- Cited by
- 431
- References
- 72
- FWCI
- 25.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.