CoChat
Sign Up
Cited by 1,425Autonomous Vehicle Technology and SafetyAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

The DOI matches its Crossref record; 268 of the 268 references it cites were found in a registry or an index; none of the 86 sources its claims cite is checked for support yet.

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Open access
Feng, Di, Haase-Schutz, Christian, Rosenbaum, Lars, Hertlein, Heinz, Glaser, Claudius, Timm, Fabian, Wiesbeck, Werner, Dietmayer, Klaus·2021·IEEE Transactions on Intelligent Transportation Systems
DOI 10.1109/tits.2020.2972974
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This review systematically summarizes deep multi-modal object detection and semantic segmentation for autonomous driving, covering sensors, datasets, fusion methodologies, and challenges. It organizes fusion around “what to fuse,” “how to fuse,” and “when to fuse,” and highlights open questions in data diversity, alignment, uncertainty, and architecture design.

Key findings

  • Most reviewed methods fuse RGB images with thermal images or LiDAR point clouds; only recently have Radar data been used, including nuScenes, Oxford Radar RobotCar, Astyx HiRes2019, and Chadwick et al.’s camera–Radar vehicle detection.
  • Dataset size ranges from 1,569 frames up to over 11 million frames; the largest labeled dataset reviewed is nuScenes with nearly 1.4M frames, and multi-modal dataset size grew by two orders of magnitude between 2014 and 2019.
  • The paper finds no conclusive evidence that one fusion method is better than others; performance is highly dependent on sensing modalities, data, and network architectures. Feature concatenation is the most common operation, especially at early and middle stages.
  • Key challenges include limited data diversity, label errors, spatial/temporal misalignment, lack of uncertainty quantification, no guideline for optimal fusion architecture, metrics focused on accuracy rather than robustness, and current networks lacking temporal cues.

Methods

  • Review/survey design: summarizes newly published multi-modal datasets (2013–2019) and fusion methodologies for deep multi-modal perception in autonomous driving, with background on sensors, test vehicles, deep object detection, and semantic segmentation.
  • Data/sample: reviews datasets including KITTI, nuScenes, ApolloScape, KAIST, Waymo Open, and others; summarizes methods in tables and reports performance/runtime comparisons on the KITTI test set.
  • Main analysis: organizes fusion by “what to fuse,” “how to fuse,” and “when to fuse”; discusses challenges and open questions in data preparation and fusion methodology; provides an interactive online platform for navigating references.

Limitations

  • Stated in the paper: there is no conclusive evidence that one fusion method is better than others; the choice of when to fuse is mainly based on intuition and empirical results; no benchmark with standard hardware or programming languages makes inference-speed comparison difficult; most reviewed methods use simple fusion operations and do not explicitly model uncertainty; few studies fuse Radar, Ultrasonic, or V2X signals; no visual analytics research exists for deep multi-modal learning.
  • Stated in the paper: common evaluation metrics summarize prediction accuracy but do not consider sensor behavior in different situations, open-set conditions, or degraded/defective sensors.
  • Evident from the text: as a review, it does not present new experimental results or primary data; its performance comparisons rely on published leaderboard or author-reported numbers, which may not be directly comparable due to different hardware and settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tits.2020.2972974

Cites

268 of 268 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 86 cited sources checked · 0 supported · 0 flagged

Show all 50 claims

  • It may state two separate claims as one. 41
  • Probably not a substantial claim. 58
  • The quote only partly states this claim. 1
85 of 85

  • The paper relies on this without support.

    Recent advancements in perception for autonomous driving are driven by deep learning.

    “Recent advancements in perception for autonomous driving are driven by deep learning.”

  • The paper relies on this without support.

    Multiple sensing modalities can be fused to exploit their complementary properties.

    “In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties.”

  • The paper relies on this without support.

    Autonomous driving requires high accuracy, robustness, and real-time performance.

    “developing deep multi-modal perception systems is especially challenging for autonomous driving because it has high requirements in accuracy, robustness, and real-time performance.”

  • The paper relies on this without support.

    A reliable perception system is prerequisite for safe driving.

    “A reliable perception system is the prerequisite for a driverless car to run safely in uncontrolled and complex driving environments.”

  • The paper relies on this without support.

    Training a deep neural network on a complex task requires a huge amount of data.

    “Training a deep neural network on a complex task requires a huge amount of data.”

  • The paper relies on this without support.

    Data quality significantly affects the performance of a deep multi-modal perception system.

    “Besides data diversity and the size of the training dataset, data quality significantly affects the performance of a deep multi-modal perception system as well.”

  • The paper relies on this without support.

    Well-calibrated sensors are prerequisite for accurate and robust multi-modal perception systems.

    “Well-calibrated sensors are the prerequisite for accurate and robust multi-modal perception systems.”

  • The paper relies on this without support.

    Explicitly modeling uncertainty or informativeness of each sensing modality is important for safe autonomous driving.

    “Explicitly modeling uncertainty or informativeness of each sensing modality is important safe autonomous driving.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reference [123] is classified as both object detection and terrain segmentation

    Minor

    The object-detection section lists [123] among works on 2D object detection, whereas Table IV describes it as 2D off-road terrain segmentation.

  • Dataset size range excludes the 500-frame Astyx dataset

    Minor

    The stated minimum is 1,569 frames, but Table II includes the Astyx HiRes2019 dataset with 500 frames.

2 of 2

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. mackowiak, 2018, CEREALS–cost-effective region-based active learning for semantic segmentation, Proc Brit Mach Vis Conf, 1
  2. sermanet, 2013, OverFeat: Integrated recognition, localization and detection using convolutional networks, Proc Int Conf Learn Representations, 1
  3. lee, 2019, Spigan: Privileged adversarial learning from simulation, Proc Int Conf Learn Representations, 1
  4. roddick, 2019, Orthographic feature transform for monocular 3D object detection, Proc Brit Mach Vis Conf, 1
  5. berger, 2014, Autonomous Driving
  6. dai, 2016, R-FCN: Object detection via region-based fully convolutional networks, Proc Adv Neural Inf Process Syst, 379
  7. 2019, PandaSet Public Large-Scale Dataset for Autonomous Driving
  8. 2019, Waymo open dataset An autonomous driving dataset
  9. kesten, 2019, Lyft level 5 av dataset 2019
  10. geyer, 2019, A2D2 AEV Autonomous Driving Dataset
  11. ramachandram, 2017, Structure optimization for deep multimodal fusion networks using graph-induced kernels, Proc 25th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
  12. valada, 2019, Self-supervised model adaptation for multimodal semantic segmentation, Int J Comput Vis, 127, 1
  13. enderich, 2019, Learning multimodal fixed-point weights using gradient descent, Proc 27th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
  14. qi, 2017, PointNet++: Deep hierarchical feature learning on point sets in a metric space, Proc Adv Neural Inf Process Syst, 5099
  15. li, 2018, PointCNN: Convolution on $\chi$ -transformed points, Proc Adv Neural Inf Process Syst, 826
  16. sauer, 2018, Conditional affordance learning for driving in urban environments, Proc 2st Annu Conf Robot Learn, 237
  17. 2017, On the Road to Fully Self-Driving Waymo Safety Report

Cited by

Sorted by citations

See all 1,425

Related

See all

Metadata

Authors
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K.
Affiliations
  • Universität Ulm (DE)
  • Karlsruhe Institute of Technology (DE)
  • Robert Bosch (Germany) (DE)
ORCID
Venue
IEEE Transactions on Intelligent Transportation Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021-03
Volume
22
Issue
3
Pages
1341-1360
ISSN
  • 1524-9050 (ISSN-L)
  • 1558-0016
DOI
10.1109/tits.2020.2972974
OpenAlex
W3008115128
Cited by
1,425
References
268
Citations by year
2026: 178 · 2025: 274 · 2024: 308 · 2023: 263 · 2022: 181 · 2021: 144
FWCI
60.51
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations
  • IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Robert Bosch GmbH
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multisensory perception, autonomous driving, object detection, semantic segmentation, sensor fusion, LiDAR, radar, camera, fusion method, onboard sensors, benchmark dataset, scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review