Cited by 731Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

The DOI matches its Crossref record; 106 of the 107 references it cites were found in a registry or an index.

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Open access

Zhang, Jiaming, Liu, Huayao, Yang, Kailun, Hu, Xinxin, Liu, Ruiping, Stiefelhagen, Rainer·2023·IEEE Transactions on Intelligent Transportation Systems

DOI 10.1109/tits.2023.3300537

Open this paper in CoChat — free account Open PDF

Abstract

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality (${X}$-modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available athttps://github.com/huaaaliu/RGBX_Semantic_Segmentation.

Integrity signals

Registry

Crossref · 10.1109/tits.2023.3300537

Cites

106 of 106 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054

Show all 107 references

Research works printed but not linked: 1

Dosovitskiy, An image is worth 16✗16 words: Transformers for image recognition at scale, 1

Sorted by citations

Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity
Tang, L., Zhang, H., Xu, H., & Ma, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101870 · Cited by 418
TruFor: Leveraging All-Round Clues for Trustworthy Image Forgery Detection and Localization
Guillaro, F., Cozzolino, D., Sud, A., Dufour, N., & Verdoliva, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01974 · Cited by 275
Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption
Liu, J., Wu, G., Liu, Z., Wang, D., Jiang, Z., Ma, L., Zhong, W., Fan, X., & Liu, R. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3521416 · Cited by 238
Self-citation
Cross-Modal Transformers for Infrared and Visible Image Fusion
Park, S. H., Vien, A. G., & Lee, C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3289170 · Cited by 219
Delivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00116 · Cited by 215
Self-citation
Context-Aware Interaction Network for RGB-T Semantic Segmentation
Lv, Y., Liu, Z., & Li, G. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3349072 · Cited by 145
WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces
Yao, S., Guan, R., Wu, Z., Ni, Y., Huang, Z., Liu, R. W., Yue, Y., Ding, W., Lim, E. G., Seo, H., et al. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3415772 · Cited by 133
Self-citation
SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking
Hou, X., Xing, J., Qian, Y., Guo, Y., Xin, S., Chen, J., Tang, K., Wang, M., Jiang, Z., Liu, L., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02507 · Cited by 143
TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection
Zhang, X., Zhang, X., Wang, J., Ying, J., Sheng, Z., Yu, H., Li, C., & Shen, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3443455 · Cited by 120
RGB-D and Thermal Sensor Fusion: A Systematic Literature Review
Brenner, M., Reyes, N. H., Sušnjak, T., & Barczak, A. L. C. (2023) · IEEE Access · doi:10.1109/access.2023.3301119 · Cited by 119
EGFNet: Edge-Aware Guidance Fusion Network for RGB–Thermal Urban Scene Parsing
Dong, S., Zhou, W., Xu, C., & Yan, W. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3306368 · Cited by 111
MRFS: Mutually Reinforcing Image Fusion and Segmentation
Zhang, H., Zuo, X., Jiang, J., Guo, C., & Ma, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02547 · Cited by 98
Misaligned Visible-Thermal Object Detection: A Drone-Based Benchmark and Baseline
Song, K., Xue, X., Wen, H., Ji, Y., Yan, Y., & Meng, Q. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3398429 · Cited by 92
Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigation
Huang, W., Zhou, Y., He, X., & Lv, C. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3312453 · Cited by 91
Transformer Fusion and Pixel-Level Contrastive Learning for RGB-D Salient Object Detection
Wu, J., Hao, F., Liang, W., & Xu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3275308 · Cited by 80
OmniVec: Learning robust representations with cross modal sharing
Srivastava, S., & Sharma, G. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00127 · Cited by 91
A Multistage Information Complementary Fusion Network Based on Flexible-Mixup for HSI-X Image Classification
Wang, J., Zhang, M., Li, W., & Tao, R. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3300903 · Cited by 74
Glass Segmentation With RGB-Thermal Image Pairs
Huo, D., Wang, J., Qian, Y., & Yang, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3256762 · Cited by 73
IF-USOD: Multimodal information fusion interactive feature enhancement architecture for underwater salient object detection
Yuan, G., Song, J., & Li, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102806 · Cited by 76
Progressive Adjacent-Layer coordination symmetric cascade network for semantic segmentation of Multimodal remote sensing images
Xiaomin, F., Zhou, W., Qian, X., & Yan, W. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121999 · Cited by 71
TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation
Liu, R., Yang, K., Roitberg, A., Zhang, J., Peng, K., Liu, H., Wang, Y., & Stiefelhagen, R. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3455416 · Cited by 88
Self-citation
DRMF: Degradation-Robust Multi-Modal Image Fusion via Composable Diffusion Prior
Tang, L., Deng, Y., Yi, X., Yan, Q., Yuan, Y., & Ma, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681064 · Cited by 69
Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding
Liu, Y., Li, C., Xu, S., & Han, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02393-8 · Cited by 59
Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58
MultiSenseSeg: A Cost-Effective Unified Multimodal Semantic Segmentation Model for Remote Sensing
Wang, Q., Chen, W., Huang, Z., Tang, H., & Yang, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390750 · Cited by 58
MMFlood: A Multimodal Dataset for Flood Delineation From Satellite Imagery
Montello, F., Arnaudo, E., & Rossi, C. (2022) · IEEE Access · doi:10.1109/access.2022.3205419 · Cited by 57
AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation
Du, S., Wang, W., Guo, R., Wang, R., & Tang, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00756 · Cited by 59
LMFNet: Lightweight Multimodal Fusion Network for high-resolution remote sensing image segmentation
Wang, T., Chen, G., Zhang, X., Liu, C., Wang, J., Tan, X., Zhou, W., & He, C. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111579 · Cited by 58
ASANet: Asymmetric Semantic Aligning Network for RGB and SAR image land cover classification
Zhang, P., Peng, B., Lu, C., Huang, Q., & Liu, D. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.09.025 · Cited by 54
DHFNet: dual-decoding hierarchical fusion network for RGB-thermal semantic segmentation
Cai, Y., Zhou, W., Zhang, L., Yu, L., & Luo, T. (2023) · The Visual Computer · doi:10.1007/s00371-023-02773-6 · Cited by 53
UTFNet: Uncertainty-Guided Trustworthy Fusion Network for RGB-Thermal Semantic Segmentation
Wang, Q., Yin, C., Song, H., Shen, T., & Gu, Y. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3322452 · Cited by 52
Removal Then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detection
Zhao, T. C., Yuan, M., Jiang, F., Wang, N., & Wei, X. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3638627 · Cited by 98
An interactive fusion attention-guided network for ground surface hot spring fluids segmentation in dual-spectrum UAV images
Yi, S., Chen, M., Yuan, X., Si, G., & Wang, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.01.022 · Cited by 52
DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation
Yin, B.-W., Cao, J.-L., Cheng, M., & Hou, Q. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01802 · Cited by 65
Transformer fusion for indoor RGB-D semantic segmentation
Wu, Z., Zhou, Z., Allibert, G., Stolz, C., Demonceaux, C., & Ma, C. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104174 · Cited by 51
IGT: Illumination-guided RGB-T object detection with transformers
Chen, K., Liu, J., & Zhang, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110423 · Cited by 50
Decoupling Common and Unique Representations for Multimodal Self-supervised Learning
Wang, Y., Albrecht, C. M., Braham, N. A. A., Liu, C., Xiong, Z., & Zhu, X. X. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73397-0_17 · Cited by 49
Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers
Zhu, Z., Hou, J., & Wu, D. O. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02015 · Cited by 52
3D-MMFN: Multi-level multimodal fusion network for 3D industrial image anomaly detection
Asad, M., Azeem, W., Malik, A. A., Jiang, H., Ali, A., Yang, J., & Liu, W. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103284 · Cited by 51
Mask-DiFuser: A Masked Diffusion Model for Unified Unsupervised Image Fusion
Tang, L., Li, C., & Ma, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3609323 · Cited by 47
A Comprehensive Review of Modern Object Segmentation Approaches
Wang, Y., Ahsan, U., Li, H., & Hagen, M. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000097 · Cited by 54
A Transformer-based multi-modal fusion network for semantic segmentation of high-resolution remote sensing imagery
Liu, Y., Gao, K., Wang, H., Yang, Z., Wang, P., Ji, S., Huang, Y., Zhu, Z., & Zhao, X. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.104083 · Cited by 43
Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning
Dong, S., Feng, Y., Yang, Q., Huang, Y., Liu, D., & Fan, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801872 · Cited by 46
MMSFormer: Multimodal Transformer for Material and Semantic Segmentation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2024) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2024.3389812 · Cited by 44
RoadFormer+: Delivering RGB-X Scene Parsing Through Scale-Aware Information Decoupling and Advanced Heterogeneous Feature Fusion
Huang, J., Li, J., Ning, J., Sun, Y., Liu, C., Chen, Q., & Fan, R. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3448251 · Cited by 46
SELMA: SEmantic Large-Scale Multimodal Acquisitions in Variable Weather, Daytime and Viewpoints
Testolina, P., Barbato, F., Michieli, U., Giordani, M., Zanuttigh, P., & Zorzi, M. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3257086 · Cited by 41
On 3D Reconstruction Using RGB-D Cameras
Tychola, K. A., Tsimperidis, I., & Papakostas, G. A. (2022) · Digital · doi:10.3390/digital2030022 · Cited by 41
U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation
Li, B., Zhang, D., Zhao, Z., Gao, J., & Li, X. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111801 · Cited by 41
FreeFusion: Infrared and Visible Image Fusion via Cross Reconstruction Learning
Zhao, W., Cui, H., Wang, H., He, Y., & Lu, H. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3572599 · Cited by 40
Complementary Random Masking for RGB-Thermal Semantic Segmentation
Shin, U., Lee, K., Kweon, I. S., & Oh, J. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611200 · Cited by 49
“The primary research direction is designing a multi-modal fusion module [44], [46], [8], [21] that can effectively combine the information from both RGB and thermal modalities to improve the accuracy of semantic segmentation.”
BackgroundMethodInfluential
CFFormer: A Cross-Fusion Transformer Framework for the Semantic Segmentation of Multisource Remote Sensing Images
Zhao, J., Zhang, M., Zhou, Z., Wang, Z., Lang, F., Shi, H., & Zheng, N. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3507274 · Cited by 46
UTLNet: Uncertainty-Aware Transformer Localization Network for RGB-Depth Mirror Segmentation
Zhou, W., Cai, Y., Zhang, L., Yan, W., & Yu, L. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3323890 · Cited by 38
IPNet: Polarization-based Camouflaged Object Detection via dual-flow network
Wang, X., Ding, J., Zhang, Z., Xu, J., & Gao, J. Y. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107303 · Cited by 38
FTransDeepLab: Multimodal Fusion Transformer-Based DeepLabv3+ for Remote Sensing Semantic Segmentation
Feng, H., Hu, Q., Zhao, P., Wang, S., Ai, M., Zheng, D., & Liu, T. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3553478 · Cited by 38
Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
Maheshwari, H., Liu, Y.-C., & Kira, Z. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00106 · Cited by 37
Graph Attention Guidance Network With Knowledge Distillation for Semantic Segmentation of Remote Sensing Images
Zhou, W., Xiaomin, F., Yan, W., Shan, S., Jiang, Q., & Hwang, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3311480 · Cited by 36
Multimodal Feature-Guided Pretraining for RGB-T Perception
Ouyang, J., Jin, P., & Wang, Q. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3454054 · Cited by 34
Feature Contrast Difference and Enhanced Network for RGB-D Indoor Scene Classification in Internet of Things
Zhou, W., Jian, B., & Liu, Y. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3537281 · Cited by 33
RFIENet: RGB-thermal feature interactive enhancement network for semantic segmentation of insulator in backlight scenes
Li, B., Wang, T., Zhai, Y., & Yuan, J. (2022) · Measurement · doi:10.1016/j.measurement.2022.112177 · Cited by 33
UnionFormer: Unified-Learning Transformer with Multi-View Representation for Image Manipulation Detection and Localization
Li, S., Ma, W., Guo, J., Xu, S., Li, B., & Zhang, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01190 · Cited by 35
TGF: Multiscale transformer graph attention network for multi-sensor image fusion
Mustafa, H. T., Shamsolmoali, P., & Lee, I. H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121789 · Cited by 32
CloudSeg: A multi-modal learning framework for robust land cover mapping under cloudy conditions
Xu, F., Shi, Y., Yang, W. T., Xia, G.-S., & Zhu, X. X. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.06.001 · Cited by 36
Difference-Complementary Learning and Label Reassignment for Multimodal Semi-Supervised Semantic Segmentation of Remote Sensing Images
Han, W., Jiang, W. R., Geng, J., & Wang, M. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3526064 · Cited by 30
CFNet: Automatic multi-modal brain tumor segmentation through hierarchical coarse-to-fine fusion and feature communication
Cheng, Y., Zheng, Y., & Wang, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106876 · Cited by 28
Light Transport Induced Domain Adaptation for Semantic Segmentation in Thermal Infrared Urban Scenes
Chen, J., Liu, Z., Jin, D., Wang, Y., Yang, F., & Bai, X. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3194931 · Cited by 28
EISNet: A Multi-Modal Fusion Network for Semantic Segmentation With Events and Images
Xie, B., Deng, Y., Shao, Z., & Li, Y. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3380255 · Cited by 30
Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera
Cao, J., Xu, Z., Lyu, Y., Wang, J., Xu, R., & Wang, L. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611705 · Cited by 35
DACFusion: Dual Asymmetric Cross-Attention guided feature fusion for multispectral object detection
Qian, J., Qiao, B., Zhang, Y., Liu, T., Wang, S., Wu, G., & Han, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129913 · Cited by 27
Exploring Multi-modal Fusion for Image Manipulation Detection and Localization
Triaridis, K., & Mezaris, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53311-2_15 · Cited by 28
A RGB-Thermal Image Segmentation Method Based on Parameter Sharing and Attention Fusion for Safe Autonomous Driving
Li, G., Lin, Y., Ouyang, D., Li, S., Luo, X., Qu, X., Pi, D., & Li, S. E. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3332350 · Cited by 28
Global feature-based multimodal semantic segmentation
Gao, S., Yang, X., Jiang, L. M., Fu, Z., & Du, J. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110340 · Cited by 26
CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes
Brödermann, T., Sakaridis, C., Fu, Y., & Gool, L. V. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3536218 · Cited by 26
Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation
Fan, J., Li, J., Hua, Z., Zhang, F., & Zhang, C. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3350593 · Cited by 25
Deep learning based object detection from multi-modal sensors: an overview
Liu, Y., Meng, S., Wang, H., & Liu, J. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16275-z · Cited by 26
DHFNet: Decoupled Hierarchical Fusion Network for RGB-T dense prediction tasks
Chen, H., Wang, Z., Qin, H., & Mu, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127594 · Cited by 24
SCAU-net: 3D self-calibrated attention U-Net for brain tumor segmentation
Liu, D., Sheng, N., Han, Y., Hou, Y., Liu, B., Zhang, J., & Zhang, Q. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08872-8 · Cited by 24
Self-citation
Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection
Cao, H., Zhang, Z., Xia, Y., Li, X., Xia, J., Chen, G., & Knoll, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72907-2_10 · Cited by 30
MGF-GCN: Multimodal interaction Mamba-aided graph convolutional fusion network for semantic segmentation of remote sensing images
Zhao, Y., Qiu, L., Yang, Z., Chen, Y., & Zhang, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103150 · Cited by 23
An Energy-Efficient, Unified CNN Accelerator for Real-Time Multi-Object Semantic Segmentation for Autonomous Vehicle
Jung, J., Kim, S., Jang, W., Seo, B., & Lee, K. J. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3349588 · Cited by 24
Polarimetric Imaging for Robot Perception: A Review
Taglione, C., Mateo, C. M., & Stolz, C. (2024) · Sensors · doi:10.3390/s24144440 · Cited by 23
UAMFDet: Acoustic‐Optical Fusion for Underwater Multi‐Modal Object Detection
Chen, H., Wang, Z., Qin, H., & Mu, X. (2024) · Journal of Field Robotics · doi:10.1002/rob.22432 · Cited by 23
Specificity-Guided Cross-Modal Feature Reconstruction for RGB-Infrared Object Detection
Sun, X., Zhu, Y., & Huang, H. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3495028 · Cited by 23
Steering Angle-Guided Multimodal Fusion Lane Detection for Autonomous Driving
Gong, Y., Zhang, X., Lu, J., Jiang, X., Wang, Z., Liu, H., Li, Z., Wang, L., Yang, Q., & Wu, X. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3507536 · Cited by 23
Self-citation
MiLNet: Multiplex Interactive Learning Network for RGB-T Semantic Segmentation
Liu, J., Liu, H., Li, X., Ren, J., & Xu, X. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3544484 · Cited by 23
Self-citation
DuSFE: Dual-Channel Squeeze-Fusion-Excitation co-attention for cross-modality registration of cardiac SPECT and CT
Chen, X., Zhou, B., Xie, H., Guo, X., Zhang, J., Duncan, J. S., Miller, E. J., Sinusas, A. J., Onofrey, J. A., & Liu, C. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102840 · Cited by 22
Self-citation
Region-adaptive and context-complementary cross modulation for RGB-T semantic segmentation
Peng, F., Ding, Z., Chen, Z., Wang, G., Hui, T., Liu, S., & Shi, H. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110092 · Cited by 23
Towards Source-Free Domain Adaptive Semantic Segmentation via Importance-Aware and Prototype-Contrast Learning
Cao, Y., Zhang, H., Lu, X. B., Xiao, Z., Yang, K., & Wang, Y. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3383157 · Cited by 22
Self-citation
SiMultiF: A Remote Sensing Multimodal Semantic Segmentation Network With Adaptive Allocation of Modal Weights for Siamese Structures in Multiscene
Cui, S., Chen, W., Xiong, W., Xu, X. J., Shi, X., & Li, C. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3553713 · Cited by 24
STSNet: A cross-spatial resolution multi-modal remote sensing deep fusion network for high resolution land-cover segmentation
Yu, B., Li, J., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102689 · Cited by 25
EMTrack: Efficient Multimodal Object Tracking
Liu, C., Guan, Z., Lai, S., Liu, Y., Lu, H., & Wang, D. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3494725 · Cited by 23
When Pedestrian Detection Meets Multi-modal Learning: Generalist Model and Benchmark Dataset
Zhang, Y., Zeng, W., Jin, S., Qian, C., Luo, P., & Liu, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73195-2_25 · Cited by 26
Late better than early: A decision-level information fusion approach for RGB-Thermal crowd counting with illumination awareness
Cheng, J., Feng, C., Xiao, Y., & Cao, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127888 · Cited by 21
MutualFormer: Multi-modal Representation Learning via Cross-Diffusion Attention
Wang, X., Wang, X., Jiang, B., Tang, J., & Luo, B. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02067-x · Cited by 29
MSMA: A multi-stage and multi-attention algorithm for the classification of multimodal skin lesions
Shu, C., Yu, L., Tian, S., & Shi, X. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106180 · Cited by 20
Understanding Dark Scenes by Contrasting Multi-Modal Observations
Dong, X., & Yokoya, N. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00089 · Cited by 22
PolyMaX: General Dense Prediction with Mask Transformer
Yang, X., Yuan, L., Wilber, M. J., Sharma, A., Gu, X., Qiao, S., Debats, S. R., Wang, H., Adam, H., Sirotenko, M., et al. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00109 · Cited by 19
Dual-modal non-local context guided multi-stage fusion for indoor RGB-D semantic segmentation
Guo, X., Ma, W., Liang, F., & Mi, Q. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124598 · Cited by 19
Image semantic segmentation of indoor scenes: A survey
Velasteguí, R., Tatarchenko, M., Karaoğlu, S., & Gevers, T. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104102 · Cited by 23
LCH: fast RGB-D salient object detection on CPU via lightweight convolutional network with hybrid knowledge distillation
Wang, B., Zhang, F., & Zhao, Y. (2023) · The Visual Computer · doi:10.1007/s00371-023-02898-8 · Cited by 18
Towards efficient RGB-T semantic segmentation via feature generative distillation strategy
Zhao, S., Wang, J., Zhang, Q., & Han, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103282 · Cited by 18

Authors: Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R.
Affiliations: Karlsruhe Institute of Technology (DE)
Nio (China) (CN)
Hunan University (CN)
ByteDance (China) (CN)
ORCID: Zhang, Jiaming
Yang, Kailun
Hu, Xinxin
Liu, Ruiping
Stiefelhagen, Rainer
Venue: IEEE Transactions on Intelligent Transportation Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2023-12
Volume: 24
Issue: 12
Pages: 14679-14694
ISSN: 1524-9050 (ISSN-L)
1558-0016
DOI: 10.1109/tits.2023.3300537
OpenAlex: W4386179772
Cited by: 731
References: 107
Citations by year: 2026: 225 · 2025: 268 · 2024: 178 · 2023: 53
FWCI: 65.71
Type: journal article
Language: en
Open access: green
arxiv.org
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: Federal Ministry of Labor and Social Affairs (BMAS) through the AccessibleMaps Project (01KM151112)
“KIT Future Fields” Project
Ministry of Science, Research and the Arts of Baden-Wurttemberg (MWK) through the Cooperative Graduate School Accessibility through AIBased Assistive Technology (BW6-03)
Federal Ministry of Education and Research (BMBF) through the Fellowship within the IFI Program of the German Academic Exchange Service
HoreKA@KIT Supercomputer Partition
Hangzhou SurImage Technology Company Ltd
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords: cross-modal fusion, multimodal fusion, depth images, thermal images, polarization images, event cameras, LiDAR, transformer models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks