Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal features can be exploited progressively. To take advantage of such scheme, we introduce two asymmetric fusion operations including channel shuffle and pixel shift, which learn different fused features with respect to different fusion directions. These two operations are parameter-free and strengthen the multimodal feature interactions across channels as well as enhance the spatial feature discrimination within channels. We conduct extensive experiments on semantic segmentation and image translation tasks, based on three publicly available datasets covering diverse modalities. Results indicate that our proposed framework is general, compact and is superior to state-of-the-art fusion frameworks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our method also performs significantly better compared to dynamic channel exchange based CEN [79] and asymmetric fusion based AsymFusion [78] models.”
From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [78]
In the citing paper
“| AsymFusion [100] | Xception65 [122] | 82.1 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [100]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“Since we have presented two asymmetric fusion operations that can be compatible with the bidirectional fusion scheme, we now consider integrating them into convolutional networks.”
BackgroundMethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“Unlike existing works which necessitate individual encoders for multiple modalities, we propose that by sharing convolutional parameters but leaving Batch Normalization layers (BNs) [16] modality-specific, we are able to train a single network for multiple modalities.”
BackgroundXception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 3×
“In group convolutions [7, 15], outputs that correspond to each channel are only related with a portion of input channels.”
BackgroundMethodEncoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 2×
“and an outdoor dataset CityScapes [8].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982 · cited 1×
“To address this issue, channel shuffle is presented in ShuffleNet [36] to enhance the information flow across different input channel groups, so that each channel is correlated with all groups.”
Method
Research works printed but not linked: 5
- Howard A.G. Zhu M. Chen B. Kalenichenko D. Wang W. Weyand T. Andreetto M. Adam H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) Howard A.G. Zhu M. Chen B. Kalenichenko D. Wang W. Weyand T. Andreetto M. Adam H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017) · arXiv
- Li Y., ICLR Workshop (2017)
- Ma P. Zhou Y. Lu Y. Zhang W.: Learning efficient video representation with video shuffle networks. arXiv preprint arXiv:1911.11319 (2019) Ma P. Zhou Y. Lu Y. Zhang W.: Learning efficient video representation with video shuffle networks. arXiv preprint arXiv:1911.11319 (2019) · arXiv
- Russakovsky O., Li, F.: Imagenet large scale visual recognition challenge. IJCV (2015)
- Yu J. Yang L. Xu N. Yang J. Huang T.S.: Slimmable neural networks. In: ICLR (2019) Yu J. Yang L. Xu N. Yang J. Huang T.S.: Slimmable neural networks. In: ICLR (2019)
Cited by
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731
“Additional methods dynamically exchange channels and shift pixels to enable implicit cross-modal feature fusion [69], [70].”
MethodDelivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00116 · Cited by 188
“To scale from modality-specific fusion to unified fusion, CMX [49] tackles RGB-X segmentation with multi-level cross-modal interactions, whereas channel-and token exchanges are explored in [72–74].”
BackgroundMethodAsymmetric Feature Fusion Network for Hyperspectral and SAR Image Classification
Li, W., Gao, Y., Zhang, M., Tao, R., & Du, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3149394 · Cited by 168
DIFNet: Boosting Visual Information Flow for Image Captioning
Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., & Ji, R. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01749 · Cited by 99
“Inspired by [30], we first share the parameters of the MHSA layer and the PWFF layer to avoid the increase of network parameters.”
BackgroundMethodRobust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
“Our method also performs significantly better compared to dynamic channel exchange based CEN [56] and asymmetric fusion based AsymFusion [57] models.”
Method
Metadata
- Authors
- Wang, Y., Sun, F., Lu, M., & Yao, A.
- Affiliations
- Tsinghua University (CN)
- Intel (United States) (US)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 3902-3910
- OpenAlex
- W3093327960
- Cited by
- 68
- References
- 33
- Citations by year
- 2026: 8 · 2025: 20 · 2024: 19 · 2023: 9 · 2022: 10 · 2021: 2
- FWCI
- 1.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation of China (NSFC) (61621136008)
- German Research Foundation (DFG) (TRR-169)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Keywords
- multimodal fusion, shared encoder, channel shuffle, pixel shift, semantic segmentation, image-to-image translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion. https://app.cochat.ai/papers/10.1145/3394171.3413621
- The paper
- Wang, Y., Sun, F., Lu, M., & Yao, A. (2020). Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3413621
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.