Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal features can be exploited progressively. To take advantage of such scheme, we introduce two asymmetric fusion operations including channel shuffle and pixel shift, which learn different fused features with respect to different fusion directions. These two operations are parameter-free and strengthen the multimodal feature interactions across channels as well as enhance the spatial feature discrimination within channels. We conduct extensive experiments on semantic segmentation and image translation tasks, based on three publicly available datasets covering diverse modalities. Results indicate that our proposed framework is general, compact and is superior to state-of-the-art fusion frameworks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our method also performs significantly better compared to dynamic channel exchange based CEN [79] and asymmetric fusion based AsymFusion [78] models.”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [78]
In the citing paper
“| AsymFusion [100] | Xception65 [122] | 82.1 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [100]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“Since we have presented two asymmetric fusion operations that can be compatible with the bidirectional fusion scheme, we now consider integrating them into convolutional networks.”
BackgroundMethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“Unlike existing works which necessitate individual encoders for multiple modalities, we propose that by sharing convolutional parameters but leaving Batch Normalization layers (BNs) [16] modality-specific, we are able to train a single network for multiple modalities.”
BackgroundXception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 3×
“In group convolutions [7, 15], outputs that correspond to each channel are only related with a portion of input channels.”
BackgroundMethodEncoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 2×
“and an outdoor dataset CityScapes [8].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982 · cited 1×
“To address this issue, channel shuffle is presented in ShuffleNet [36] to enhance the information flow across different input channel groups, so that each channel is correlated with all groups.”
MethodRethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · cited 3×
“In [29], Markov random fields are explored for indoor segmentation based on RGB and depth inputs.”
BackgroundRefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287 · cited 1×
“RDFNet [17] adopts multi-layer fusion at four downsampling stages of the ResNet (encoder), iteratively refines the fused features with the similar idea in RefineNet [22], and then sends these fused features to the decoder.”
MethodEigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · cited 1×
“For example, [9, 10] stack multimodal inputs by channel-wise concatenation and then feed them to the network.”
BackgroundTSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,055 · cited 1×
“Also in [23], a temporal shift module is proposed to shift a portion of channels along the temporal dimension, aiming to boost information exchange among neighboring frames.”
BackgroundLearning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528
Taskonomy: Disentangling Task Transfer Learning
Zamir, A. R., Sax, A., Shen, W., Guibas, L., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00391 · Cited by 1,128
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · cited 4×
“For example, regarding to the semantic segmentation task, FuseNet [13], RDFNet [17], SSMA [28] adopt two equal-sized encoders for RGB and depth inputs respectively.”
BackgroundMethodInfluentialAdaptive batch normalization for practical domain adaptation
Li, Y., Wang, N., Shi, J., Hou, X., & Liu, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.03.005 · Cited by 601
Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images
Gupta, S., Arbelaez, P., & Malik, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.79 · Cited by 595 · cited 3×
“[11] improves RGB-D recognition performance by making use of the constructed geometric contour from depth data.”
BackgroundMethodShift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J., & Keutzer, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00951 · Cited by 381
Indoor Semantic Segmentation using depth information
Couprie, C., Farabet, C., Najman, L. A., & LeCun, Y. (2013) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1301.3572 · Cited by 372 · cited 1×
“For example, [9, 10] stack multimodal inputs by channel-wise concatenation and then feed them to the network.”
BackgroundSearching for Efficient Multi-Scale Architectures for Dense Image Prediction
Chen, L.-C., Collins, M. D., Zhu, Y., Papandreou, G., Zoph, B., Schroff, F., Adam, H., & Shlens, J. (2018) · arXiv · doi:10.48550/arxiv.1809.04184 · Cited by 347
In-place Activated BatchNorm for Memory-Optimized Training of DNNs
Bulo, S. R., Porzi, L., & Kontschieder, P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00591 · Cited by 320
RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation
Lee, S., Park, S.-J., & Hong, K.-S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.533 · Cited by 285 · cited 5×
“Recently, to enable more powerful feature alignment, some multi-layer fusion works adopt a pile of 3 × 3 convolutional layers [17] or attention-based designs [28] for fusion.”
BackgroundMethodInfluentialLocality-Sensitive Deconvolution Networks with Gated Fusion for RGB-D Indoor Semantic Segmentation
Cheng, Y., Cai, R., Li, Z., Zhao, X., & Huang, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.161 · Cited by 229 · cited 1×
“[6, 24] explore multimodal feature fusion at the prediction side.”
BackgroundRefineNet: Multi-Path Refinement Networks for Dense Prediction
Lin, G., Liu, F., Milan, A., Shen, C., & Reid, I. D. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2893630 · Cited by 172 · cited 1×
“For semantic segmentation tasks, in order to facilitate comparison with previous fusion approaches, we choose RefineNet [21] for the dataset NYUDv2 and DeepLabv3+ [4] for the dataset Cityscapes, with backbone architectures ResNet [14] and Xception65 [7] respectively.”
MethodLearning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Wang, J., Wang, Z., Tao, D., See, S., & Wang, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_40 · Cited by 161
Cascaded Feature Network for Semantic Segmentation of RGB-D Images
Lin, D., Chen, G., Cohen-Or, D., Heng, P.-A., & Huang, H. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.147 · Cited by 149
Chen, W., Xie, D., Zhang, Y., & Pu, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00741 · Cited by 91 · cited 1×
“The shift design is further improved in [5], where only a few shift operations are needed,”
BackgroundSCN: Switchable Context Network for Semantic Segmentation of RGB-D Images
Lin, D., Zhang, R., Ji, Y., Li, P., & Huang, H. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2885062 · Cited by 87 · cited 1×
“Methods to exploit multimodal information have been studied for decades [11–13, 20, 29], which allow better understanding of visual scenes compared to learning with unimodal inputs.”
BackgroundK for the Price of 1: Parameter-efficient Multi-task and Transfer LearningPreprint
Mudrakarta, P. K., Sandler, M., Zhmoginov, A., & Howard, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10703 · Cited by 77 · cited 1×
“Sharing network parameters but privatizing BNs has been proved to be effective for efficient model adaption when considering multiple tasks or multiple domains [18, 26, 31, 33].”
BackgroundDeep Surface Normal Estimation With Hierarchical RGB-D Fusion
Zeng, J., Tong, Y., Huang, Y., Yan, Q., Sun, W., Chen, J., & Wang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00631 · Cited by 69 · cited 3×
“[35] also adopts two branches for learning RGB-D features respectively, and merges depth features into RGB branch at all downsampling layers in a hierarchical manner.”
BackgroundSelf-citationZhuang, Y., Yang, F., Tao, L., Ma, C., Zhang, Z., Li, Y., Jia, H., Xie, X., & Gao, W. (2018) · 2018 25th IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2018.8451830 · Cited by 41
Resolution Switchable Networks for Runtime Efficient Image Recognition
Wang, Y., Sun, F., Li, D., & Yao, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_32 · Cited by 37
Self-citation
Research works printed but not linked: 5
- Howard A.G. Zhu M. Chen B. Kalenichenko D. Wang W. Weyand T. Andreetto M. Adam H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) Howard A.G. Zhu M. Chen B. Kalenichenko D. Wang W. Weyand T. Andreetto M. Adam H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) · arXiv
- Li Y., ICLR Workshop (2017)
- Ma P. Zhou Y. Lu Y. Zhang W.: Learning efficient video representation with video shuffle networks. arXiv preprint arXiv:1911.11319 (2019) Ma P. Zhou Y. Lu Y. Zhang W.: Learning efficient video representation with video shuffle networks. arXiv preprint arXiv:1911.11319 (2019) · arXiv
- Russakovsky O., Li, F.: Imagenet large scale visual recognition challenge. IJCV (2015)
- Yu J. Yang L. Xu N. Yang J. Huang T.S.: Slimmable neural networks. In: ICLR (2019) Yu J. Yang L. Xu N. Yang J. Huang T.S.: Slimmable neural networks. In: ICLR (2019)
Cited by
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731
“Additional methods dynamically exchange channels and shift pixels to enable implicit cross-modal feature fusion [69], [70].”
MethodDelivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00116 · Cited by 188
“To scale from modality-specific fusion to unified fusion, CMX [49] tackles RGB-X segmentation with multi-level cross-modal interactions, whereas channel-and token exchanges are explored in [72–74].”
BackgroundMethodAsymmetric Feature Fusion Network for Hyperspectral and SAR Image Classification
Li, W., Gao, Y., Zhang, M., Tao, R., & Du, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3149394 · Cited by 168
DIFNet: Boosting Visual Information Flow for Image Captioning
Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., & Ji, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01749 · Cited by 99
“Inspired by [30], we first share the parameters of the MHSA layer and the PWFF layer to avoid the increase of network parameters.”
BackgroundMethodRobust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
“Our method also performs significantly better compared to dynamic channel exchange based CEN [56] and asymmetric fusion based AsymFusion [57] models.”
Method
Metadata
- Authors
- Wang, Y., Sun, F., Lu, M., & Yao, A.
- Affiliations
- Tsinghua University (CN)
- Intel (United States) (US)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 3902-3910
- OpenAlex
- W3093327960
- Cited by
- 68
- References
- 33
- Citations by year
- 2026: 8 · 2025: 20 · 2024: 19 · 2023: 9 · 2022: 10 · 2021: 2
- FWCI
- 1.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation of China (NSFC) (61621136008)
- German Research Foundation (DFG) (TRR-169)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Keywords
- multimodal fusion, shared encoder, channel shuffle, pixel shift, semantic segmentation, image-to-image translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.