Cited by 1,425Autonomous Vehicle Technology and SafetyAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

The DOI matches its Crossref record; 268 of the 268 references it cites were found in a registry or an index; none of the 86 sources its claims cite is checked for support yet.

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Open access

Feng, Di, Haase-Schutz, Christian, Rosenbaum, Lars, Hertlein, Heinz, Glaser, Claudius, Timm, Fabian, Wiesbeck, Werner, Dietmayer, Klaus·2021·IEEE Transactions on Intelligent Transportation Systems

DOI 10.1109/tits.2020.2972974

Open this paper in CoChat — free account Open PDF

Abstract

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

AI snapshot

Written from the paper’s full text.

TL;DR

This review systematically summarizes deep multi-modal object detection and semantic segmentation for autonomous driving, covering sensors, datasets, fusion methodologies, and challenges. It organizes fusion around “what to fuse,” “how to fuse,” and “when to fuse,” and highlights open questions in data diversity, alignment, uncertainty, and architecture design.

Key findings

Most reviewed methods fuse RGB images with thermal images or LiDAR point clouds; only recently have Radar data been used, including nuScenes, Oxford Radar RobotCar, Astyx HiRes2019, and Chadwick et al.’s camera–Radar vehicle detection.
Dataset size ranges from 1,569 frames up to over 11 million frames; the largest labeled dataset reviewed is nuScenes with nearly 1.4M frames, and multi-modal dataset size grew by two orders of magnitude between 2014 and 2019.
The paper finds no conclusive evidence that one fusion method is better than others; performance is highly dependent on sensing modalities, data, and network architectures. Feature concatenation is the most common operation, especially at early and middle stages.
Key challenges include limited data diversity, label errors, spatial/temporal misalignment, lack of uncertainty quantification, no guideline for optimal fusion architecture, metrics focused on accuracy rather than robustness, and current networks lacking temporal cues.

Methods

Review/survey design: summarizes newly published multi-modal datasets (2013–2019) and fusion methodologies for deep multi-modal perception in autonomous driving, with background on sensors, test vehicles, deep object detection, and semantic segmentation.
Data/sample: reviews datasets including KITTI, nuScenes, ApolloScape, KAIST, Waymo Open, and others; summarizes methods in tables and reports performance/runtime comparisons on the KITTI test set.
Main analysis: organizes fusion by “what to fuse,” “how to fuse,” and “when to fuse”; discusses challenges and open questions in data preparation and fusion methodology; provides an interactive online platform for navigating references.

Limitations

Stated in the paper: there is no conclusive evidence that one fusion method is better than others; the choice of when to fuse is mainly based on intuition and empirical results; no benchmark with standard hardware or programming languages makes inference-speed comparison difficult; most reviewed methods use simple fusion operations and do not explicitly model uncertainty; few studies fuse Radar, Ultrasonic, or V2X signals; no visual analytics research exists for deep multi-modal learning.
Stated in the paper: common evaluation metrics summarize prediction accuracy but do not consider sensor behavior in different situations, open-set conditions, or degraded/defective sensors.
Evident from the text: as a review, it does not present new experimental results or primary data; its performance comparisons rely on published leaderboard or author-reported numbers, which may not be directly comparable due to different hardware and settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tits.2020.2972974

Cites

268 of 268 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 86 cited sources checked · 0 supported · 0 flagged

None of the sources cited for this claim have been checked yet.
In the paper
“Only the very recently nuScenes [89], Oxford Radar RobotCar [85] and Astyx HiRes2019 Datasets [94] provide Radar data.”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [89] [89] H. Caesar et al. , 'nuScenes: A multimodal dataset for autonomous driving,' arXiv preprint arXiv:1903.11027 , 2019. -
- Not checked yetcites [85] [85] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and I. Posner, 'The Oxford radar RobotCar dataset: A radar extension to the Oxford RobotCar dataset,' arXiv preprint arXiv: 1909.01300 , 2019. [Online]. Available: https://arxiv.org/pdf/1909.01300 - source
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [94] [94] M. Meyer and G. Kuschk, 'Automotive radar dataset for deep learning based 3d object detection,' in Proceedings of the 16th European Radar Conference , 2019. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim have been checked yet.
In the paper
“Most of the reviewed datasets provide ground-truth labels for 2D object detection and semantic segmentation tasks [60], [75], [88], [90]-[93].”
What the sources say
- Not checked yetcites [60] [60] X. Huang et al. , 'The ApolloScape dataset for autonomous driving,' in Workshop Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018, pp. 954-960. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
- Not checked yetcites [88] [88] M. Braun, S. Krebs, F. B. Flohr, and D. M. Gavrila, 'EuroCity Persons: A novel benchmark for person detection in traffic scenes,' IEEE Trans. Pattern Anal. Mach. Intell. , pp. 1-1, 2019. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In this regard, some papers including [98], [102]-[105], [107], [113], [115] combine RGB camera images and LiDAR point clouds for 3D object detection.”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [98] [98] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, 'Multi-view 3D object detection network for autonomous driving,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2017, pp. 6526-6534. -
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [102]-[105]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [107] [107] X. Du, M. H. Ang Jr., S. Karaman, and D. Rus, 'A general pipeline for 3d detection of vehicles,' in IEEE Int. Conf. Robotics and Automation , 2018. -
Strength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim have been checked yet.
In the paper
“The size of open multi-modal datasets is usually much smaller than the size of image datasets.”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. -
- Not checked yetcites [163] [163] O. Russakovsky et al. , 'ImageNet large scale visual recognition challenge,' Int. J. Computer Vision , vol. 115, no. 3, pp. 211-252, 2015. -
Load-bearingStrength:plain· not yet calibratedn = 1,034,908
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
In the paper
“While the interest in processing Radar signals via deep learning is growing, only a few papers discuss deep multi-modal perception with Radar for autonomous driving (e.g. [134]).”
What the sources say
- Not checked yetcites [134] [134] S. Chadwick, W. Maddern, and P. Newman, 'Distant vehicle detection using radar and vision,' in IEEE Int. Conf. Robotics and Automation , 2019. -
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
None of the sources cited for this claim have been checked yet.
In the paper
“Many works [61], [91], [99][101], [106], [108], [109], [111], [117]-[120], [123] deal with the 2D object detection problem on the front-view 2D image plane.”
What the sources say
- Not checked yetcites [61] [61] L. Schneider et al. , 'Multimodal neural networks: RGB-D for semantic segmentation and object detection,' in Scandinavian Conf. Image Analysis . Springer, 2017, pp. 98-109. - source
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [91] [91] K. Takumi, K. Watanabe, Q. Ha, A. Tejero-De-Pablos, Y. Ushiku, and T. Harada, 'Multispectral object detection for autonomous vehicles,' in Proc. Thematic Workshops of ACM Multimedia , 2017, pp. 35-43. -
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [99][101]
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim have been checked yet.
In the paper
“The network is much more robust against labeling errors when they are randomly distributed, compared to biased labeling from the use of a deterministic pre-labeling.”
What the sources say
- Not checked yetcites [194] [194] C. Haase-Sch¨ utz, H. Hertlein, and W. Wiesbeck, 'Estimating labeling quality with deep object detectors,' in IEEE Intelligent Vehicles Symp. , June 2019, pp. 33-38. -
- Not checked yetcites [195] [195] S. Chadwick and P. Newman, 'Training object detectors with noisy data,' in IEEE Intelligent Vehicles Symp. , June 2019, pp. 1319-1325. -
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim have been checked yet.
In the paper
“Compared to the object detection problem summarized in Sec. IV-A, there are fewer works on multi-modal semantic segmentation: [92], [119], [124] employ RGB and thermal images, [61] fuses RGB images and depth images from a stereo camera, [125]-[127] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [123] fuses RGB images and LiDAR point clouds”
What the sources say
- Not checked yetcites [92] [92] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, 'MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2017, pp. 5108-5115. - source
- Not checked yetcites [119] [119] D. Guan, Y. Cao, J. Liang, Y. Cao, and M. Y. Yang, 'Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection,' Information Fusion , vol. 50, pp. 148-157, 2019. - source
- Not checked yetcites [124] [124] Y. Sun, W. Zuo, and M. Liu, 'RTFNet: Rgb-thermal fusion network for semantic segmentation of urban scenes,' IEEE Robotics and Automation Letters , 2019. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“Most existing works combine RGB images from visual cameras with 3D LiDAR point clouds [98]-[116].”
What the sources say
- Not checked yetcites [98]-[116]
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes

Show all 50 claims

It may state two separate claims as one. 41
Probably not a substantial claim. 58
The quote only partly states this claim. 1

In the paper
“Even though the KITTI dataset [75] is widely used for autonomous driving research, the diversity of its recording conditions is relatively low: it is recorded in Karlsruhe -a mid-sized city in Germany, only during daytime and on sunny days.”
What the sources say
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“The recent Eurocity dataset [88] is the most diverse dataset we have reviewed. It is recorded in different cities from several European countries. All seasons are considered, as well as weather and daytime diversity. To date, the dataset is camera-only and other modalities (e.g. LiDARs) are announced.”
What the sources say
- Not checked yetcites [88] [88] M. Braun, S. Krebs, F. B. Flohr, and D. M. Gavrila, 'EuroCity Persons: A novel benchmark for person detection in traffic scenes,' IEEE Trans. Pattern Anal. Mach. Intell. , pp. 1-1, 2019. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“The dataset size ranges from only 1,569 frames up to over 11 million frames. The largest dataset with ground-truth labels that we have reviewed is the nuScenes Dataset [89] with nearly 1,4M frames.”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [89] [89] H. Caesar et al. , 'nuScenes: A multimodal dataset for autonomous driving,' arXiv preprint arXiv:1903.11027 , 2019. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“From the tables we have three observations: (1). Most works propose to fuse LiDAR and camera features extracted from 2D convolution neural networks. To do this, they project LiDAR points on the 2D plane and process the feature maps through 2D convolutions. Only a few works extract LiDAR features by PointNet (e.g. [104], [105], [128]) or 3D convolutions (e.g. [123]); (2). Several works on multi”
What the sources say
- Not checked yetcites [104] [104] D. Xu, D. Anguelov, and A. Jain, 'PointFusion: Deep sensor fusion for 3D bounding box estimation,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source
- Not checked yetcites [105] [105] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum PointNets for 3d object detection from RGB-D data,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source
- Not checked yetcites [128] [128] F. Yang, J. Yang, Z. Jin, and H. Wang, 'A fusion model for road detection based on deep learning and fully connected CRF,' in 13th Annu. Conf. System of Systems Engineering . IEEE, 2018, pp. 29-36. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one. The quote only partly states this claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Some other works focus on fusing the RGB images from visual cameras with images from thermal cameras [91], [117]-[119].”
What the sources say
- Not checked yetcites [91] [91] K. Takumi, K. Watanabe, Q. Ha, A. Tejero-De-Pablos, Y. Ushiku, and T. Harada, 'Multispectral object detection for autonomous vehicles,' in Proc. Thematic Workshops of ACM Multimedia , 2017, pp. 35-43. -
- Not checked yetcites [117]-[119]
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A deep neural network is a powerful tool for learning hierarchical feature representations given a large amount of data [5].”
What the sources say
- Not checked yetcites [5] [5] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, p. 436, 2015. - source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A spherical map is obtained by projecting each 3D point onto a sphere, characterized by azimuth and zenith angles. It has the advantage of representing each 3D point in a dense and compact way, making it a suitable representation for point cloud segmentation (e.g. [51]). However, the size of the representation can be different from camera images. Therefore, it is difficult to fuse them at an early”
What the sources say
- Not checked yetcites [51] [51] B. Wu, A. Wan, X. Yue, and K. Keutzer, 'SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3d lidar point cloud,' in IEEE Int. Conf. Robotics and Automation , May 2018, pp. 1887-1893. - source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“650 scans (dense), 34200 scans (sparse)”
What the sources say
- Not checked yetcites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“95 , 328 frames, 103 , 128 objects”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Visual and thermal camera”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“2015”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Car, Van, Truck, Pedestrian, Person (sitting), Cyclist, Tram, Misc”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Seoul”
What the sources say
- Not checked yetcites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“7481 frames (training) 80 . 256 objects”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Visual camera, LiDAR, and GNSS”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“No dynamic objects”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Visual (Stereo) camera, 3D LiDAR, GNSS, and inertial sensors”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Fukuoka”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“2D bounding box”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“2014”
What the sources say
- Not checked yetcites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“2012, 2013, 2015”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Person, People, Cyclist during day and night”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Karlsruhe”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Place categorization”
What the sources say
- Not checked yetcites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Visual (Stereo) camera, 5 × 2D LiDAR (yielding 3D information), GNSS and inertial sensors”
What the sources say
- Not checked yetcites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“no”
What the sources say
- Not checked yetcites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source
Strength:plain· not yet calibrated
Tests✗ Asserted: fails✓ Checkable: passes✗ Load-bearing: fails✗ Based: fails
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“2D/3D bounding box, visual odometry, road, optical flow, tracking, depth, 2D instance and pixel-level segmentation”
What the sources say
- Not checked yetcites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source
- Not checked yetcites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“2016”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“M´ alaga”
What the sources say
- Not checked yetcites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“n.a.”
What the sources say
- Not checked yetcites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Qi et al. , 2017 [105] | 70.39 % | 81.20 % | 62.19 % | 44.89 % | 51.21 % | 40.23 % | 56.77 % | 71.96 % | 50.39 % | 0.17 s | GPU @ 3.0 Ghz (Python) |”
What the sources say
- Not checked yetcites [105] [105] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum PointNets for 3d object detection from RGB-D data,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Liang et al. , 2018 [150] | 66.22 % | 82.54 % | 64.04 % | - | - | - | - | - | - | 0.06 s | GPU @ 2.5 Ghz (Python) |”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [150] [150] M. Liang, B. Yang, S. Wang, and R. Urtasun, 'Deep continuous fusion for multi-sensor 3d object detection,' in Proc. Eur. Conf. Computer Vision , 2018, pp. 641-656. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Sindagi et al. , 2019 [113] | 72.7 % | 83.2 % | 65.12 % | - | - | - | - | - | - | - | - |”
What the sources say
- Not checked yetcites [113] [113] V. A. Sindagi, Y. Zhou, and O. Tuzel, 'MVX-Net: Multimodal voxelnet for 3D object detection,' in IEEE Int. Conf. Robotics and Automation , 2019. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Ku et al. , 2017 [103] | 71.88 % | 81.94 % | 66.38 % | 42.81 % | 50.80 % | 40.88 % | 52.18 % | 64.00 % | 46.61 % | 0.1 s | GPU Titan X (Pascal) |”
What the sources say
- Not checked yetcites [103] [103] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander, 'Joint 3d proposal generation and object detection from view aggregation,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , Oct. 2018, pp. 1-8. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Liang et al. , 2019 [116] | 76.75 % | 86.81 % | 68.41 % | 45.61 % | 52.37 % | 41.49 % | 64.68 % | 79.58 % | 57.03 % | 0.08 s | GPU @ 2.5 Ghz (Python) |”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [116] [116] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun, 'Multi-task multisensor fusion for 3D object detection,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2019, pp. 7345-7353. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Qi et al. , 2017 [105] | 70.39 % | 81.20 % | 62.19 % | 44.89 % | 51.21 % | 40.23 % | 56.77 % | 71.96 % | 50.39 % | 0.17 s | GPU @ 3.0 Ghz (Python) |”
What the sources say
- Not checked yetcites [105] [105] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum PointNets for 3d object detection from RGB-D data,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Shin et al. , 2018 [141] | 73.04 % | 83.71 % | 59.16 % | - | - | - | - | - | - | - | GPU Titan X (not Pascal) |”
What the sources say
- Not checked yetcites [141] [141] K. Shin, Y. P. Kwon, and M. Tomizuka, 'RoarNet: A robust 3D object detection based on region approximation refinement,' in IEEE Intelligent Vehicles Symp. , 2018. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“113 , 082frames, 5 , 654 . 6s (camera); > 220 , 000frames, 5 , 000s (LiDARs)”
What the sources say
- Not checked yetcites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Liang et al. , 2019 [116] | 76.75 % | 86.81 % | 68.41 % | 45.61 % | 52.37 % | 41.49 % | 64.68 % | 79.58 % | 57.03 % | 0.08 s | GPU @ 2.5 Ghz (Python) |”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [116] [116] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun, 'Multi-task multisensor fusion for 3D object detection,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2019, pp. 7345-7353. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Ku et al. , 2017 [103] | 71.88 % | 81.94 % | 66.38 % | 42.81 % | 50.80 % | 40.88 % | 52.18 % | 64.00 % | 46.61 % | 0.1 s | GPU Titan X (Pascal) |”
What the sources say
- Not checked yetcites [103] [103] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander, 'Joint 3d proposal generation and object detection from view aggregation,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , Oct. 2018, pp. 1-8. - source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Chen et al. , 2017 [98] | 62.35 % | 71.09 % | 55.12 % | - | - | - | - | - | - | 0.36 s | GPU @ 2.5 Ghz (Python + C/C++) |”
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [98] [98] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, 'Multi-view 3D object detection network for autonomous driving,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2017, pp. 6526-6534. -
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.

85 of 85

The paper relies on this without support.
Recent advancements in perception for autonomous driving are driven by deep learning.
“Recent advancements in perception for autonomous driving are driven by deep learning.”
The paper relies on this without support.
Multiple sensing modalities can be fused to exploit their complementary properties.
“In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties.”
The paper relies on this without support.
Autonomous driving requires high accuracy, robustness, and real-time performance.
“developing deep multi-modal perception systems is especially challenging for autonomous driving because it has high requirements in accuracy, robustness, and real-time performance.”
The paper relies on this without support.
A reliable perception system is prerequisite for safe driving.
“A reliable perception system is the prerequisite for a driverless car to run safely in uncontrolled and complex driving environments.”
The paper relies on this without support.
Training a deep neural network on a complex task requires a huge amount of data.
“Training a deep neural network on a complex task requires a huge amount of data.”
The paper relies on this without support.
Data quality significantly affects the performance of a deep multi-modal perception system.
“Besides data diversity and the size of the training dataset, data quality significantly affects the performance of a deep multi-modal perception system as well.”
The paper relies on this without support.
Well-calibrated sensors are prerequisite for accurate and robust multi-modal perception systems.
“Well-calibrated sensors are the prerequisite for accurate and robust multi-modal perception systems.”
The paper relies on this without support.
Explicitly modeling uncertainty or informativeness of each sensing modality is important for safe autonomous driving.
“Explicitly modeling uncertainty or informativeness of each sensing modality is important safe autonomous driving.”

8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Reference [123] is classified as both object detection and terrain segmentation
Minor
The object-detection section lists [123] among works on 2D object detection, whereas Table IV describes it as 2D off-road terrain segmentation.
Dataset size range excludes the 500-frame Astyx dataset
Minor
The stated minimum is 1,569 frames, but Table II includes the Astyx HiRes2019 dataset with 500 frames.

2 of 2

Items 201–268 of 268

From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation NetworkPreprint
Shi, S., Wang, Z., Shi, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03670 · Cited by 99
Choosing smartly: Adaptive multimodal fusion for object detection in changing environments
Mees, O., Eitel, A., & Burgard, W. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759048 · Cited by 96
Automotive radar gridmap representations
Werber, K., Rapp, M., Klappstein, J., Hahn, M., Dickmann, J., Dietmayer, K., & Waldschmidt, C. (2015) · IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM) · doi:10.1109/icmim.2015.7117922 · Cited by 88
Self-citation
Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector
Feng, D., Wei, X., Rosenbaum, L., Maki, A., & Dietmayer, K. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8814236 · Cited by 85
Self-citation
Semisupervised and Weakly Supervised Road Detection Based on Generative Adversarial Networks
Han, X., Lu, J., Zhao, C., You, S., & Li, H. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2809685 · Cited by 85
Car detection for autonomous vehicle: LIDAR and vision fusion approach through deep learning framework
Du, X., Ang, M. H., & Rus, D. L. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202234 · Cited by 83
Comparison of random forest and long short-term memory network performances in classification tasks using radar
Schumann, O., Wöhler, C., Hahn, M., & Dickmann, J. (2017) · Sensor Data Fusion: Trends, Solutions, Applications (SDF) · doi:10.1109/sdf.2017.8126350 · Cited by 83
Leveraging Heteroscedastic Aleatoric Uncertainties for Robust Real-Time LiDAR 3D Object Detection
Feng, D., Rosenbaum, L., Timm, F., & Dietmayer, K. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814046 · Cited by 82
Self-citation
Robust Deep Multi-modal Learning Based on Gated Information Fusion Network
Kim, J., Koh, J., Kim, Y., Choi, J., Hwang, Y., & Choi, J. W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_6 · Cited by 82
Fusing Bird’s Eye View LIDAR Point Cloud and Front View Camera Image for 3D Object Detection
Wang, Z., Zhan, W., & Tomizuka, M. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500387 · Cited by 81
Tracking Multiple Vehicles Using a Variational Radar Model
Scheel, A., & Dietmayer, K. (2018) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2018.2879041 · Cited by 79
Self-citation
When to use what data set for your self-driving car algorithm: An overview of publicly available driving datasets
Yin, H., & Berger, C. (2017) · 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317828 · Cited by 79
SynthCity: A large scale synthetic point cloudPreprint
Griffiths, D., & Boehm, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04758 · Cited by 79
Boosting LiDAR-Based Semantic Labeling by Cross-modal Training Data Generation
Piewak, F., Pinggera, P., Schäfer, M., Peter, D., Schwarz, B., Schneider, N., Enzweiler, M., Pfeiffer, D., & Zöllner, M. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11024-6_39 · Cited by 78
Semantic Segmentation of 3D LiDAR Data in Dynamic Scene Using Semi-Supervised Learning
Mei, J., Gao, B., Xu, D., Yao, W., Zhao, X., & Zhao, H. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2919741 · Cited by 78
On the Importance of Label Quality for Semantic Segmentation
Zlateski, A., Jaroensri, R., Sharma, P., & Durand, F. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00160 · Cited by 77
Uncertainty Estimation for Deep Neural Object Detectors in Safety-Critical Applications
Le, M. T., Diehl, F., Brunner, T., & Knoll, A. (2018) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569637 · Cited by 77
DepthCN: Vehicle detection using 3D-LIDAR and ConvNet
Asvadi, A., Garrote, L., Premebida, C., Peixoto, P., & Nunes, U. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317880 · Cited by 77
Potential of radar for static object classification using deep learning methods
Lombacher, J., Hahn, M., Dickmann, J., & Wohler, C. (2016) · 2016 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM) · doi:10.1109/icmim.2016.7533931 · Cited by 76
Deeply-Fused NetsPreprint
Wang, J., Wei, Z., Zhang, T., & Zeng, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07716 · Cited by 73
Deep stochastic radar models
Wheeler, T. A., Holder, M., Winner, H., & Kochenderfer, M. J. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995697 · Cited by 73
Deep active learning for object detection
Roy, S. D., Unmesh, A., & Namboodiri, V. P. (2018) · Cited by 71
A Cross-Season Correspondence Dataset for Robust Semantic Segmentation
Mans Larsson, M., Stenborg, E., Hammarstrand, L., Pollefeys, M., Sattler, T., & Kahl, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00976 · Cited by 71
3-D LiDAR + Monocular Camera: An Inverse-Depth-Induced Fusion Framework for Urban Road Detection
Gu, S., Lu, T., Zhang, Y., Álvarez, J. M., Yang, J. J., & Kong, H. (2018) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2018.2843170 · Cited by 71
Box2Pix: Single-Shot Instance Segmentation by Assigning Pixels to Object Boxes
Uhrig, J., Rehder, E., Fröhlich, B., Franke, U., & Brox, T. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500621 · Cited by 68
High-resolution LIDAR-based depth mapping using bilateral filter
Premebida, C., Garrote, L., Asvadi, A., Ribeiro, A. P., & Nunes, U. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795953 · Cited by 68
Learning the Structure of Deep Convolutional Networks
Feng, J., & Darrell, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.315 · Cited by 64
BLVD: Building A Large-scale 5D Semantics Benchmark for Autonomous Driving
Xue, J., Fang, J., Li, T., Zhang, B., Zhang, P., Ye, Z., & Dou, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793523 · Cited by 62
Deep semantic classification for 3D LiDAR data
Dewan, A., Oliveira, G. L., & Burgard, W. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206198 · Cited by 60
Semantic radar grids
Lombacher, J., Laudt, K., Hahn, M., et al. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995871 · Cited by 59
Learning End-to-end Multimodal Sensor Policies for Autonomous NavigationPreprint
Liu, G.-H., Siravuru, A., Prabhakar, S., Veloso, M., & Kantor, G. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10422 · Cited by 58
Optimal Sensor Data Fusion Architecture for Object Detection in Adverse Weather Conditions
Pfeuffer, A., & Dietmayer, K. (2018) · International Conference on Information Fusion (FUSION) · doi:10.23919/icif.2018.8455757 · Cited by 54
Self-citation
SEG-VoxelNet for 3D Vehicle Detection from RGB and LiDAR Data
Dou, J., Xue, J., & Fang, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793492 · Cited by 54
A LiDAR Point Cloud Generator: from a Virtual World to Autonomous DrivingPreprint
Yue, X., Wu, B., Seshia, S. A., Keutzer, K., & Sangiovanni-Vincentelli, A. L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.00103 · Cited by 51
Early Fusion of Camera and Lidar for robust road detection based on U-Net FCN
Wulff, F., Schäufele, B., Sawade, O., Becker, D., Henke, B., & Radusch, I. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500549 · Cited by 49
Blockout: Dynamic Model Selection for Hierarchical Deep Networks
Murdock, C., Li, Z., Zhou, H., & Duerig, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.283 · Cited by 48
Combining LiDAR space clustering and convolutional neural networks for pedestrian detection
Matti, D., Ekenel, H. K., & Thiran, J. (2017) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2017.8078512 · Cited by 48
Training Object Detectors With Noisy Data
Chadwick, S., & Newman, P. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8814137 · Cited by 46
Robust detection of non-motorized road users using deep learning on optical and LIDAR data
Kim, T., & Ghosh, J. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795566 · Cited by 42
Multimodal Neural Networks: RGB-D for Semantic Segmentation and Object Detection
Schneider, L., Jasch, M., Fröhlich, B., Weber, T., Franke, U., Pollefeys, M., & Rätsch, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-59126-1_9 · Cited by 42
Moving Target Classification in Automotive Radar Systems Using Convolutional Recurrent Neural Networks
Kim, S., Lee, S., Doo, S., & Shim, B. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553185 · Cited by 40
Capturing Object Detection Uncertainty in Multi-Layer Grid Maps
Wirges, S., Reith‐Braun, M., Lauer, M., et al. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814073 · Cited by 37
Leveraging Pre-Trained 3D Object Detection Models for Fast Ground Truth Generation
Lee, J., Walsh, S., Harakeh, A., & Waslander, S. L. (2018) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569793 · Cited by 34
Semantic Grid Estimation with a Hybrid Bayesian and Deep Neural Network Approach
Erkent, Ö., Wolf, C., Laugier, C., González, D., & Romero-Cano, V. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593434 · Cited by 34
Seeing Through Fog Without Seeing Fog: Deep Sensor Fusion in the Absence of Labeled Training Data.Preprint
Bijelic, M., Mannan, F., Gruber, T., Ritter, W., Dietmayer, K., & Heide, F. (2019) · arXiv (Cornell University) · Cited by 32
Self-citation
Multi-modal sensor registration for vehicle perception via deep neural networks
Giering, M. J., Venugopalan, V., & Reddy, K. (2015) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec.2015.7322485 · Cited by 31
The Well-Calibrated Bayesian: Rejoinder
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.2307/2287723 · Cited by 30
Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous DrivingPreprint
Feng, D., Rosenbaum, L., Glaeser, C., Timm, F., & Dietmayer, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12358 · Cited by 28
Self-citation
Part-A^2 Net: 3D Part-Aware and Aggregation Neural Network for Object Detection from Point CloudPreprint
Shi, S., Wang, Z., Wang, X., et al. (2019) · arXiv (Cornell University) · Cited by 25
Season-Invariant Semantic Segmentation with a Deep Multimodal Network
Kim, D.-K., Maturana, D., Uenoyama, M., & Scherer, S. (2017) · Springer proceedings in advanced robotics · doi:10.1007/978-3-319-67361-5_17 · Cited by 24
DeepTemporalSeg: Temporally Consistent Semantic Segmentation of 3D LiDAR Scans
Dewan, A., & Burgard, W. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197193 · Cited by 23
A Novel Approach for Detecting Road Based on Two-Stream Fusion Fully Convolutional Network
Lv, X., Liu, Z., Xin, J., & Zheng, N. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500551 · Cited by 21
Robust Drivable Road Region Detection for Fixed-Route Autonomous Vehicles Using Map-Fusion Images
Cai, Y., Li, D., Zhou, X. L., et al. (2018) · Sensors · doi:10.3390/s18124158 · Cited by 21
Stereo Ground Truth with Error Bars
Kondermann, D., Nair, R. S., Meister, S., Mischler, W., Güssefeld, B., Honauer, K., Hofmann, S., Claus Brenner, A., & Jähne, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16814-2_39 · Cited by 19
Understanding deep neural networks performance for radar-based human motion recognition
Amin, M. G., & Erol, B. (2018) · IEEE Radar Conference (RadarConf) · doi:10.1109/radar.2018.8378780 · Cited by 18
Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal
Wang, Z., & Jia, K. (2019) · IEEE Conference Proceedings · Cited by 17
Multi-modal panoramic 3D outdoor datasets for place categorization
Jung, H., Oto, Y., Mozos, O. M., Iwashita, Y., & Kurazume, R. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759669 · Cited by 16
Learning the Structure of Deep Architectures Using L1 Regularization
Kulkarni, P., Zepeda, J., Jurie, F., Pérez, P., & Chevallier, L. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.23 · Cited by 16
Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object DetectionPreprint
Wang, Z., Zhan, W., & Tomizuka, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06703 · Cited by 16
From Footprints to Beliefprints: Motion Planning under Uncertainty for Maneuvering Automated Vehicles in Dense Scenarios
Banzhaf, H., Dolgov, M., Stellet, J. E., et al. (2018) · International Conference on Intelligent Transportation Systems · doi:10.1109/itsc.2018.8569897 · Cited by 14
A*3D Dataset: Towards Autonomous Driving in Challenging EnvironmentsPreprint
Pham, Q.-H., Sevestre, P., Pahwa, R. S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07541 · Cited by 14
Modout: Learning Multi-Modal Architectures by Stochastic Regularization
Li, F., Neverova, N., Wolf, C., & Taylor, G. W. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.59 · Cited by 13
Classification of objects in polarimetric radar images using CNNs at 77 GHz
Visentin, T., Sagainov, A., Hasch, J., & Zwick, T. (2017) · Asia-Pacific Microwave Conference (APMC) · doi:10.1109/apmc.2017.8251453 · Cited by 13
A Fusion Model for Road Detection based on Deep Learning and Fully Connected CRF
Yang, F., Yang, J., Jin, Z., & Wang, H. (2018) · 2018 13th Annual Conference on System of Systems Engineering (SoSE) · doi:10.1109/sysose.2018.8428696 · Cited by 13
Multi-camera traffic light recognition using a classifying Labeled Multi-Bernoulli filter
Bach, M., Reuter, S., & Dietmayer, K. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995852 · Cited by 12
Self-citation
Probability-based Detection Quality (PDQ): A Probabilistic Approach to Detection Evaluation.Preprint
Hall, D. R., Dayoub, F., Skinner, J., et al. (2018) · arXiv (Cornell University) · Cited by 11
On Boosting Semantic Street Scene Segmentation with Weak Supervision
Meletis, P., & Dubbelman, G. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814217 · Cited by 8
Estimating Labeling Quality with Deep Object Detectors
Haase-Schütz, C., Hertlein, H., & Wiesbeck, W. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8814144 · Cited by 8
Self-citation

Flagged first, then by citations

Research works printed but not linked: 17

mackowiak, 2018, CEREALS–cost-effective region-based active learning for semantic segmentation, Proc Brit Mach Vis Conf, 1
sermanet, 2013, OverFeat: Integrated recognition, localization and detection using convolutional networks, Proc Int Conf Learn Representations, 1
lee, 2019, Spigan: Privileged adversarial learning from simulation, Proc Int Conf Learn Representations, 1
roddick, 2019, Orthographic feature transform for monocular 3D object detection, Proc Brit Mach Vis Conf, 1
berger, 2014, Autonomous Driving
dai, 2016, R-FCN: Object detection via region-based fully convolutional networks, Proc Adv Neural Inf Process Syst, 379
2019, PandaSet Public Large-Scale Dataset for Autonomous Driving
2019, Waymo open dataset An autonomous driving dataset
kesten, 2019, Lyft level 5 av dataset 2019
geyer, 2019, A2D2 AEV Autonomous Driving Dataset
ramachandram, 2017, Structure optimization for deep multimodal fusion networks using graph-induced kernels, Proc 25th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
valada, 2019, Self-supervised model adaptation for multimodal semantic segmentation, Int J Comput Vis, 127, 1
enderich, 2019, Learning multimodal fixed-point weights using gradient descent, Proc 27th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
qi, 2017, PointNet++: Deep hierarchical feature learning on point sets in a metric space, Proc Adv Neural Inf Process Syst, 5099
li, 2018, PointCNN: Convolution on $\chi$ -transformed points, Proc Adv Neural Inf Process Syst, 826
sauer, 2018, Conditional affordance learning for driving in urban environments, Proc 2st Annu Conf Robot Learn, 237
2017, On the Road to Fully Self-Driving Waymo Safety Report

Sorted by citations

YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00721 · Cited by 13,198
“For example, multi-object tracking [94, 93], autonomous driving [40, 18], robotics [35, 58], medical image analysis [34, 46], etc.”
Background
YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2022) · arXiv · doi:10.48550/arxiv.2207.02696 · Cited by 13,205
“For example, multi-object tracking [94, 93], autonomous driving [40, 18], robotics [35, 58], medical image analysis [34, 46], etc.”
Background
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
A Survey of Autonomous Driving: Common Practices and Emerging Technologies
Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. (2020) · IEEE Access · doi:10.1109/access.2020.2983149 · Cited by 1,889
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072

See all 1,425

Authors: Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K.
Affiliations: Universität Ulm (DE)
Karlsruhe Institute of Technology (DE)
Robert Bosch (Germany) (DE)
ORCID: Feng, Di
Haase-Schutz, Christian
Glaser, Claudius
Wiesbeck, Werner
Dietmayer, Klaus
Venue: IEEE Transactions on Intelligent Transportation Systems
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2021-03
Volume: 22
Issue: 3
Pages: 1341-1360
ISSN: 1524-9050 (ISSN-L)
1558-0016
DOI: 10.1109/tits.2020.2972974
OpenAlex: W3008115128
Cited by: 1,425
References: 268
Citations by year: 2026: 178 · 2025: 274 · 2024: 308 · 2023: 263 · 2022: 181 · 2021: 144
FWCI: 60.51
Type: journal article
Language: en
Open access: green
arxiv.org
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations: IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: Robert Bosch GmbH
Topics: Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords: multisensory perception, autonomous driving, object detection, semantic segmentation, sensor fusion, LiDAR, radar, camera, fusion method, onboard sensors, benchmark dataset, scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks