Cited by 731Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

The DOI matches its Crossref record; 106 of the 107 references it cites were found in a registry or an index.

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Open access

Zhang, Jiaming, Liu, Huayao, Yang, Kailun, Hu, Xinxin, Liu, Ruiping, Stiefelhagen, Rainer·2023·IEEE Transactions on Intelligent Transportation Systems

DOI 10.1109/tits.2023.3300537

Open this paper in CoChat — free account Open PDF

Abstract

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality (${X}$-modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available athttps://github.com/huaaaliu/RGBX_Semantic_Segmentation.

Integrity signals

Registry

Crossref · 10.1109/tits.2023.3300537

Cites

106 of 106 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
Pyramid Scene Parsing Network
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
Non-local Neural Networks
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Dual Attention Network for Scene Segmentation
Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., & Lu, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00326 · Cited by 7,351
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Deep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1×
“Recently, SETR [31] and Segmenter [32] directly adopt vision transformers [21], [22] as the backbone, which captures global context from very early layers.”
Background
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R. A., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,369
Unified Perceptual Parsing for Scene Understanding
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_26 · Cited by 2,201 · cited 2×
“Specifically, in addition to MiT backbones, we experiment with DeepLabV3+ [120] and Swin transformer [24] backbones with UperNet [121] to construct CMX.”
Method
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.667 · Cited by 2,089
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,514
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · arXiv · doi:10.48550/arxiv.1604.01685 · Cited by 1,174
Joint 2D-3D-Semantic Data for Indoor Scene UnderstandingPreprint
Armeni, I., (Sasha), A. S., Zamir, A., & Savarese, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01105 · Cited by 929
Learning a Discriminative Feature Network for Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00199 · Cited by 909
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902
Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
Takikawa, T., Acuna, D., Jampani, V., & Fidler, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00533 · Cited by 896
CGNet: A Light-Weight Context Guided Network for Semantic Segmentation
Wu, T., Tang, S., Zhang, R., Cao, J., & Zhang, Y. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3042065 · Cited by 859
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
Liao, Y., Xie, J., & Geiger, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3179507 · Cited by 770
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Prakash, A., Chitta, K., & Geiger, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00700 · Cited by 660
Fast-SCNN: Fast Semantic Segmentation NetworkPreprint
Poudel, R. P. K., Liwicki, S., & Cipolla, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04502 · Cited by 617
Exploring Cross-Image Pixel Contrast for Semantic Segmentation
Wang, W., Zhou, T., Yu, F., Dai, J., Konukoglu, E., & Gool, L. V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00721 · Cited by 604
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
Pohlen, T., Hermans, A., Mathias, M., & Leibe, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.353 · Cited by 600
PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing
Xu, D., Ouyang, W., Wang, X., & Sebe, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00077 · Cited by 551
Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes
Sun, Y., Zuo, W., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2904733 · Cited by 546
ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation
Hu, X., Yang, K., Fei, L., & Wang, K. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803025 · Cited by 500
Self-citation
Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation
Chen, X., Lin, K.-Y., Wang, J., Wu, W., Qian, C., Li, H., & Zeng, G. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_33 · Cited by 488
3D Graph Neural Networks for RGBD Semantic Segmentation
Qi, X., Liao, R., Jia, J., Fidler, S., & Urtasun, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.556 · Cited by 482
In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images
Oršić, M., Krešo, I., Bevandić, P., & Šegvić, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01289 · Cited by 440
Rethinking Semantic Segmentation: A Prototype View
Zhou, T., Wang, W., Konukoğlu, E., & Goo, L. V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00261 · Cited by 426
3DMV: Joint 3D-Multi-view Prediction for 3D Semantic Scene Segmentation
Dai, A., & Nießner, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_28 · Cited by 352
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation
Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00423 · Cited by 352
GMNet: Graded-Feature Multilabel-Learning Network for RGB-Thermal Urban Scene Semantic Segmentation
Zhou, W., Liu, J., Lei, J., Yu, L., & Hwang, J.-N. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3109518 · Cited by 344
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis
Seichter, D., Kohler, M., Lewandowski, B., Wengefeld, T., & Gross, H.-M. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561675 · Cited by 330
ACFNet: Attentional Class Feature Network for Semantic Segmentation
Zhang, F., Chen, Y., Li, Z., Hong, Z., Liu, J., Ma, F., Han, J., & Ding, E. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00690 · Cited by 326
Depth-Aware CNN for RGB-D Segmentation
Wang, W., & Neumann, U. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_9 · Cited by 314
Efficient Attention: Attention with Linear Complexities
Shen, Z., Zhang, M., Zhao, H., Yi, S., & Li, H. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00357 · Cited by 309
RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation
Lee, S., Park, S., & Hong, K.-S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.533 · Cited by 294
Self-supervised model adaptation for multimodal semantic segmentation
Valada, A., Mohan, R., & Burgard, W. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01188-y · Cited by 293
PST900: RGB-Thermal Calibration, Dataset and Segmentation Network
Shivakumar, S. S., Rodrigues, N., Zhou, A., Miller, I. D., Kumar, V., & Taylor, C. J. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196831 · Cited by 273
Multimodal Token Fusion for Vision Transformers
Wang, Y., Chen, X., Cao, L., Huang, W., Sun, F., & Wang, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01187 · Cited by 272
FuseSeg: Semantic segmentation of urban scenes based on RGB and thermal data fusion
Sun, Y., Zuo, W., Yun, P., Wang, H., & Liu, M. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2993143 · Cited by 270
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268
CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection
Pang, Y., Zhao, X., Zhang, L., & Lu, H. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3234702 · Cited by 246
Locality-Sensitive Deconvolution Networks with Gated Fusion for RGB-D Indoor Semantic Segmentation
Cheng, Y., Cai, R., Li, Z., Zhao, X., & Huang, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.161 · Cited by 244
Omnivore: A Single Model for Many Visual Modalities
Girdhar, R., Singh, M., Ravi, N., van der Maaten, L. J. P., Joulin, A., & Misra, I. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01563 · Cited by 235
ABMDRNet: Adaptive-weighted Bi-directional Modality Difference Reduction Network for RGB-T Semantic Segmentation
Zhang, Q., Zhao, S., Luo, Y., Zhang, D., Huang, N., & Han, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00266 · Cited by 225
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01597 · Cited by 216
ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation
Cao, J., Leng, H., Lischinski, D., Cohen-Or, D., Tu, C., & Li, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00700 · Cited by 216
Combining Events and Frames Using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction
Gehrig, D., Rüegg, M., Gehrig, M., Hidalgo‐Carrió, J., & Scaramuzza, D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3060707 · Cited by 182
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182
Real-Time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-Driving Images
Sun, L., Yang, K., Hu, X., Hu, W., & Wang, K. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3007457 · Cited by 178
Self-citation
HRFormer: High-Resolution Transformer for Dense Prediction
Yuan, Y., Fu, R., Huang, L., Lin, W., Zhang, C., Chen, X., & Wang, J. (2021) · arXiv · doi:10.48550/arxiv.2110.09408 · Cited by 178
Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation
Chen, L.-Z., Lin, Z., Wang, Z., Yang, Y., & Cheng, M.-M. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3049332 · Cited by 177
FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation
Deng, F., Feng, H., Liang, M., Wang, H., Yang, Y., Gao, Y., Chen, J., Hu, J., Guo, X., & Lam, T. L. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636084 · Cited by 170
EV-SegNet: Semantic Segmentation for Event-Based Cameras
Alonso, Í., & Murillo, A. C. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00205 · Cited by 168
Cascaded Feature Network for Semantic Segmentation of RGB-D Images
Lin, D., Chen, G., Cohen‐Or, D., Heng, P., & Huang, H. F. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.147 · Cited by 155
Deep Polarization Cues for Transparent Object Segmentation
Kalra, A., Taamazyan, V., Rao, S. K., Venkataraman, K., Raskar, R., & Kadambi, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00863 · Cited by 152
CANet: Co-attention network for RGB-D semantic segmentation
Zhou, H., Qi, L., Huang, H. Y., Xu, Y., Wan, Z., & Wen, X. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108468 · Cited by 132
Bidirectional Projection Network for Cross Dimension Scene Understanding
Hu, W., Zhao, H., Jiang, L., Jia, J., & Wong, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01414 · Cited by 125
Recurrent Scene Parsing with Perspective Understanding in the Loop
Kong, S., & Fowlkes, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00106 · Cited by 114
ESS: Learning Event-Based Semantic Segmentation from Still Images
Sun, Z., Messikommer, N., Gehrig, D., & Scaramuzza, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_20 · Cited by 113
Polarization-driven semantic segmentation via efficient attention-bridged fusion
Xiang, K., Yang, K., & Wang, K. (2021) · Optics Express · doi:10.1364/oe.416130 · Cited by 111
Self-citation
Non-Local Aggregation for RGB-D Semantic Segmentation
Zhang, G., Xue, J., Xie, P., Yang, S., & Wang, G. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3066071 · Cited by 103
Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving
Zhou, W., Berrío, J. S., Worrall, S., & Nebot, E. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2909066 · Cited by 102
Attention fusion network for multi-spectral semantic segmentation
Xu, J., Lu, K., & Wang, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.015 · Cited by 99
Trans4Trans: Efficient Transformer for Transparent Object and Semantic Scene Segmentation in Real-World Navigation Assistance
Zhang, J., Yang, K., Constantinescu, A., Peng, K., Müller, K., & Stiefelhagen, R. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3161141 · Cited by 98
Self-citation
Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World
Zhang, J., Yang, K., Constantinescu, A., Peng, K., Müller, K., & Stiefelhagen, R. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00202 · Cited by 88
Self-citation
RGB-D Semantic Segmentation and Label-Oriented Voxelgrid Fusion for Accurate 3D Semantic Mapping
Shi, W., Xu, J., Zhu, D., Zhang, G., Wang, X., Li, J., & Zhang, X. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3056726 · Cited by 73
ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data
Zhang, J., Yang, K., & Stiefelhagen, R. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636109 · Cited by 69
Self-citation
Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
Wang, Y., Sun, F., Lu, M., & Yao, A. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413621 · Cited by 68
Omnisupervised Omnidirectional Semantic Segmentation
Yang, K., Hu, X., Fang, Y., Wang, K., & Stiefelhagen, R. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3023331 · Cited by 60
Self-citation
StructToken: Rethinking Semantic Segmentation With Structural Prior
Lin, F., Liang, Z., Wu, S., He, J., Chen, K., & Tian, S. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3252807 · Cited by 60
Hyperfusion-Net: Hyper-densely reflective feature fusion for salient object detection
Zhang, P., Liu, W., Lei, Y., & Lu, H. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.05.012 · Cited by 54
DHFNet: dual-decoding hierarchical fusion network for RGB-thermal semantic segmentation
Cai, Y., Zhou, W., Zhang, L., Yu, L., & Luo, T. (2023) · The Visual Computer · doi:10.1007/s00371-023-02773-6 · Cited by 53
HRFuser: A Multi-Resolution Sensor Fusion Architecture for 2D Object Detection
Brödermann, T., Sakaridis, C., Dai, D., & Gool, L. V. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422432 · Cited by 46
Malleable 2.5D Convolution: Learning Receptive Fields Along the Depth-Axis for RGB-D Scene Parsing
Xing, Y., Wang, J., & Zeng, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_33 · Cited by 45
Two-Stage Cascaded Decoder for Semantic Segmentation of RGB-D Images
Yue, Y., Zhou, W., Lei, J., & Yu, L. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3084855 · Cited by 43
Exploring Event-Driven Dynamic Context for Accident Scene Segmentation
Zhang, J., Yang, K., & Stiefelhagen, R. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3134828 · Cited by 38
Self-citation
Semantic Segmentation by Early Region Proxy
Zhang, Y., Pang, B., & Lu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00132 · Cited by 35
Gated-Residual Block for Semantic Segmentation Using RGB-D Data
Qian, Y., Deng, L., Li, T., Wang, C., & Yang, M. Z. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3107672 · Cited by 32
NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation across RGB-Depth, Polarization, and Thermal Images
Yan, R., Yang, K., & Wang, K. (2021) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio54168.2021.9739390 · Cited by 29
Self-citation
Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth ImagesPreprint
Fooladgar, F., & Shohreh Kasaei, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11691 · Cited by 25
A multimodal vision sensor for autonomous driving
Sun, D., Huang, X. M., & Yang, K. (2019) · Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies · doi:10.1117/12.2535552 · Cited by 23
Self-citation
Depth-Adapted CNN for RGB-D Cameras
Wu, Z., Allibert, G., Stolz, C., & Demonceaux, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69538-5_24 · Cited by 23
ConvMLP: Hierarchical Convolutional MLPs for VisionPreprint
Li, J., Hassani, A., Walton, S., & Shi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04454 · Cited by 23
Multilevel Cross-Aware RGBD Indoor Semantic Segmentation for Bionic Binocular Robot
Shi, W., Zhu, D., Zhang, G., Xu, J., Wang, X., Chen, L., Li, J., & Zhang, X. (2020) · IEEE Transactions on Medical Robotics and Bionics · doi:10.1109/tmrb.2020.3009527 · Cited by 14
Real-time Pedestrian Localization and State Estimation Using Moving Horizon Estimation
Mohammadbagher, E., Bhatt, N. P., Hashemi, E., Fi̇dan, B., & Khajepour, A. (2020) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294306 · Cited by 12
Tube-Embedded Transformer for Pixel Prediction
Zhang, X., Zhang, S., Cui, Z., Li, Z., Xie, J., & Yang, J. J. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3147664 · Cited by 10

Flagged first, then by citations

1 of the 107 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

Dosovitskiy, An image is worth 16✗16 words: Transformers for image recognition at scale, 1

Sorted by citations

Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity
Tang, L., Zhang, H., Xu, H., & Ma, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101870 · Cited by 418
TruFor: Leveraging All-Round Clues for Trustworthy Image Forgery Detection and Localization
Guillaro, F., Cozzolino, D., Sud, A., Dufour, N., & Verdoliva, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01974 · Cited by 275
Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption
Liu, J., Wu, G., Liu, Z., Wang, D., Jiang, Z., Ma, L., Zhong, W., Fan, X., & Liu, R. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3521416 · Cited by 238
Self-citation
Cross-Modal Transformers for Infrared and Visible Image Fusion
Park, S. H., Vien, A. G., & Lee, C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3289170 · Cited by 219
Delivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00116 · Cited by 215
Self-citation

See all 731

Authors: Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R.
Affiliations: Karlsruhe Institute of Technology (DE)
Nio (China) (CN)
Hunan University (CN)
ByteDance (China) (CN)
ORCID: Zhang, Jiaming
Yang, Kailun
Hu, Xinxin
Liu, Ruiping
Stiefelhagen, Rainer
Venue: IEEE Transactions on Intelligent Transportation Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2023-12
Volume: 24
Issue: 12
Pages: 14679-14694
ISSN: 1524-9050 (ISSN-L)
1558-0016
DOI: 10.1109/tits.2023.3300537
OpenAlex: W4386179772
Cited by: 731
References: 107
Citations by year: 2026: 225 · 2025: 268 · 2024: 178 · 2023: 53
FWCI: 65.71
Type: journal article
Language: en
Open access: green
arxiv.org
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: Federal Ministry of Labor and Social Affairs (BMAS) through the AccessibleMaps Project (01KM151112)
“KIT Future Fields” Project
Ministry of Science, Research and the Arts of Baden-Wurttemberg (MWK) through the Cooperative Graduate School Accessibility through AIBased Assistive Technology (BW6-03)
Federal Ministry of Education and Research (BMBF) through the Fellowship within the IFI Program of the German Academic Exchange Service
HoreKA@KIT Supercomputer Partition
Hangzhou SurImage Technology Company Ltd
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords: cross-modal fusion, multimodal fusion, depth images, thermal images, polarization images, event cameras, LiDAR, transformer models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks