CoChat
Sign Up
Cited by 731Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

The DOI matches its Crossref record; 106 of the 107 references it cites were found in a registry or an index.

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Open access
Zhang, Jiaming, Liu, Huayao, Yang, Kailun, Hu, Xinxin, Liu, Ruiping, Stiefelhagen, Rainer·2023·IEEE Transactions on Intelligent Transportation Systems
DOI 10.1109/tits.2023.3300537
Open this paper in CoChat — free accountOpen PDF

Abstract

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality (${X}$-modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available athttps://github.com/huaaaliu/RGBX_Semantic_Segmentation.

Overview

Integrity signals

Registry

Crossref · 10.1109/tits.2023.3300537

Cites

106 of 106 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 107 references

Research works printed but not linked: 1

  1. Dosovitskiy, An image is worth 16✗16 words: Transformers for image recognition at scale, 1

Cited by

Sorted by citations

Items 601–700 of 731

Related

See all

Metadata

Authors
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R.
Affiliations
  • Karlsruhe Institute of Technology (DE)
  • Nio (China) (CN)
  • Hunan University (CN)
  • ByteDance (China) (CN)
ORCID
Venue
IEEE Transactions on Intelligent Transportation Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023-12
Volume
24
Issue
12
Pages
14679-14694
ISSN
  • 1524-9050 (ISSN-L)
  • 1558-0016
DOI
10.1109/tits.2023.3300537
OpenAlex
W4386179772
Cited by
731
References
107
Citations by year
2026: 225 · 2025: 268 · 2024: 178 · 2023: 53
FWCI
65.71
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Federal Ministry of Labor and Social Affairs (BMAS) through the AccessibleMaps Project (01KM151112)
  • “KIT Future Fields” Project
  • Ministry of Science, Research and the Arts of Baden-Wurttemberg (MWK) through the Cooperative Graduate School Accessibility through AIBased Assistive Technology (BW6-03)
  • Federal Ministry of Education and Research (BMBF) through the Fellowship within the IFI Program of the German Academic Exchange Service
  • HoreKA@KIT Supercomputer Partition
  • Hangzhou SurImage Technology Company Ltd
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
cross-modal fusion, multimodal fusion, depth images, thermal images, polarization images, event cameras, LiDAR, transformer models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review