CoChat
Sign Up
Cited by 1,425Autonomous Vehicle Technology and SafetyAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Open access
Feng, Di, Haase-Schutz, Christian, Rosenbaum, Lars, Hertlein, Heinz, Glaser, Claudius, Timm, Fabian, Wiesbeck, Werner, Dietmayer, Klaus·2021·IEEE Transactions on Intelligent Transportation Systems
DOI 10.1109/tits.2020.2972974
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

AI snapshot

Written from the paper’s full text.

TL;DR

This review systematically summarizes deep multi-modal object detection and semantic segmentation for autonomous driving, covering sensors, datasets, fusion methodologies, and challenges. It organizes fusion around “what to fuse,” “how to fuse,” and “when to fuse,” and highlights open questions in data diversity, alignment, uncertainty, and architecture design.

Key findings

  • Most reviewed methods fuse RGB images with thermal images or LiDAR point clouds; only recently have Radar data been used, including nuScenes, Oxford Radar RobotCar, Astyx HiRes2019, and Chadwick et al.’s camera–Radar vehicle detection.
  • Dataset size ranges from 1,569 frames up to over 11 million frames; the largest labeled dataset reviewed is nuScenes with nearly 1.4M frames, and multi-modal dataset size grew by two orders of magnitude between 2014 and 2019.
  • The paper finds no conclusive evidence that one fusion method is better than others; performance is highly dependent on sensing modalities, data, and network architectures. Feature concatenation is the most common operation, especially at early and middle stages.
  • Key challenges include limited data diversity, label errors, spatial/temporal misalignment, lack of uncertainty quantification, no guideline for optimal fusion architecture, metrics focused on accuracy rather than robustness, and current networks lacking temporal cues.

Methods

  • Review/survey design: summarizes newly published multi-modal datasets (2013–2019) and fusion methodologies for deep multi-modal perception in autonomous driving, with background on sensors, test vehicles, deep object detection, and semantic segmentation.
  • Data/sample: reviews datasets including KITTI, nuScenes, ApolloScape, KAIST, Waymo Open, and others; summarizes methods in tables and reports performance/runtime comparisons on the KITTI test set.
  • Main analysis: organizes fusion by “what to fuse,” “how to fuse,” and “when to fuse”; discusses challenges and open questions in data preparation and fusion methodology; provides an interactive online platform for navigating references.

Limitations

  • Stated in the paper: there is no conclusive evidence that one fusion method is better than others; the choice of when to fuse is mainly based on intuition and empirical results; no benchmark with standard hardware or programming languages makes inference-speed comparison difficult; most reviewed methods use simple fusion operations and do not explicitly model uncertainty; few studies fuse Radar, Ultrasonic, or V2X signals; no visual analytics research exists for deep multi-modal learning.
  • Stated in the paper: common evaluation metrics summarize prediction accuracy but do not consider sensor behavior in different situations, open-set conditions, or degraded/defective sensors.
  • Evident from the text: as a review, it does not present new experimental results or primary data; its performance comparisons rely on published leaderboard or author-reported numbers, which may not be directly comparable due to different hardware and settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tits.2020.2972974

Cites

266 of 266 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 86 cited sources checked · 0 supported · 0 flagged

  • Findingcitation
    § A. Sensing Modalities · p. 4

    Only a few reviewed datasets provide Radar data: nuScenes, Oxford Radar RobotCar, and Astyx HiRes2019.

    “Only the very recently nuScenes [89], Oxford Radar RobotCar [85] and Astyx HiRes2019 Datasets [94] provide Radar data.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [89] [89] H. Caesar et al. , 'nuScenes: A multimodal dataset for autonomous driving,' arXiv preprint arXiv:1903.11027 , 2019. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [85] [85] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and I. Posner, 'The Oxford radar RobotCar dataset: A radar extension to the Oxford RobotCar dataset,' arXiv preprint arXiv: 1909.01300 , 2019. [Online]. Available: https://arxiv.org/pdf/1909.01300 - source

      Not checked in the free run.

    • cites [94] [94] M. Meyer and G. Kuschk, 'Automotive radar dataset for deep learning based 3d object detection,' in Proceedings of the 16th European Radar Conference , 2019. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation
    § D. Labels · p. 5

    Most reviewed datasets provide ground-truth labels for 2D object detection and semantic segmentation tasks.

    “Most of the reviewed datasets provide ground-truth labels for 2D object detection and semantic segmentation tasks [60], [75], [88], [90]-[93].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [60] [60] X. Huang et al. , 'The ApolloScape dataset for autonomous driving,' in Workshop Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018, pp. 954-960. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

    • cites [88] [88] M. Braun, S. Krebs, F. B. Flohr, and D. M. Gavrila, 'EuroCity Persons: A novel benchmark for person detection in traffic scenes,' IEEE Trans. Pattern Anal. Mach. Intell. , pp. 1-1, 2019. - source

      Not checked in the free run.

  • Findingown data
    § A. Multi-modal Data Preparation · p. 11

    The distribution of objects is very imbalanced, with much more objects being labeled as car than person or cyclist.

    “The distribution of objects is also very imbalanced, with much more objects being labeled as car than person or cyclist (Fig. 5).”
    Quote in the paperSubstantial claim
  • Findingcitation
    § A. Deep Multi-modal Object Detection · p. 5

    Some papers combine RGB camera images and LiDAR point clouds for 3D object detection.

    “In this regard, some papers including [98], [102]-[105], [107], [113], [115] combine RGB camera images and LiDAR point clouds for 3D object detection.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [98] [98] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, 'Multi-view 3D object detection network for autonomous driving,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2017, pp. 6526-6534. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [102]-[105]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [107] [107] X. Du, M. H. Ang Jr., S. Karaman, and D. Rus, 'A general pipeline for 3d detection of vehicles,' in IEEE Int. Conf. Robotics and Automation , 2018. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingreasoning
    § A. Multi-modal Data Preparation · p. 11

    The size of open multi-modal datasets is usually much smaller than the size of image datasets.

    “The size of open multi-modal datasets is usually much smaller than the size of image datasets.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. -

      Not checked in the free run.

    • cites [163] [163] O. Russakovsky et al. , 'ImageNet large scale visual recognition challenge,' Int. J. Computer Vision , vol. 115, no. 3, pp. 211-252, 2015. -

      Not checked in the free run.

  • May combine two claims. 41
  • Classified as probably not a substantial claim. 58
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data
    § C. Dataset Size · p. 5

    The multi-modal datasets are still relatively small compared to image datasets in the computer vision community, but the dataset size has grown by two orders of magnitude between 2014 and 2019.

    “Compared to the image datasets in the computer vision community, the multi-modal datasets are still relatively small. However, the dataset size has grown by two orders of magnitudes between 2014 and 2019 (cf. Fig. 5(b)).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation
    § B. Recording Conditions · p. 5

    The KITTI dataset is widely used for autonomous driving research, but its recording conditions are relatively low in diversity: it is recorded in Karlsruhe, only during daytime and on sunny days.

    “Even though the KITTI dataset [75] is widely used for autonomous driving research, the diversity of its recording conditions is relatively low: it is recorded in Karlsruhe -a mid-sized city in Germany, only during daytime and on sunny days.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation
    § I. INTRODUCTION · p. 1

    All methods with the highest performance on the KITTI bird's eye view test dataset for car detection are based on deep learning, and many methods that fuse camera and LiDAR information produce better performance than those using either LiDAR or camera alone.

    “All methods with the highest performance are based on deep learning, and many methods that fuse camera and LiDAR information produce better performance than those using either LiDAR or camera alone.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § A. Sensing Modalities · p. 4

    All reviewed datasets include RGB camera images.

    “All reviewed datasets include RGB camera images.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation
    § B. Recording Conditions · p. 5

    The recent Eurocity dataset is the most diverse dataset reviewed, recorded in different cities from several European countries, considering all seasons, weather and daytime diversity; to date it is camera-only and other modalities are announced.

    “The recent Eurocity dataset [88] is the most diverse dataset we have reviewed. It is recorded in different cities from several European countries. All seasons are considered, as well as weather and daytime diversity. To date, the dataset is camera-only and other modalities (e.g. LiDARs) are announced.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [88] [88] M. Braun, S. Krebs, F. B. Flohr, and D. M. Gavrila, 'EuroCity Persons: A novel benchmark for person detection in traffic scenes,' IEEE Trans. Pattern Anal. Mach. Intell. , pp. 1-1, 2019. -

      Not checked in the free run.

  • May combine two claims.

    Findingcitation
    § C. Dataset Size · p. 5

    The dataset size ranges from 1,569 frames up to over 11 million frames; the largest dataset with ground-truth labels reviewed is nuScenes with nearly 1.4M frames.

    “The dataset size ranges from only 1,569 frames up to over 11 million frames. The largest dataset with ground-truth labels that we have reviewed is the nuScenes Dataset [89] with nearly 1,4M frames.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [89] [89] H. Caesar et al. , 'nuScenes: A multimodal dataset for autonomous driving,' arXiv preprint arXiv:1903.11027 , 2019. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § A. Related Works · p. 2

    Many new multi-modal datasets have been released since 2016.

    “However, many new multi-modal datasets have been released since 2016, and it is worth summarizing them.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § I. INTRODUCTION · p. 1

    Many methods have been proposed that employ deep learning to fuse multi-modal sensors for scene understanding in autonomous driving.

    “In this regard, many methods have been proposed that employ deep learning to fuse multi-modal sensors for scene understanding in autonomous driving.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § A. What to Fuse · p. 7

    From the tables, most works propose to fuse LiDAR and camera features extracted from 2D convolution neural networks; several works cluster and segment 3D LiDAR points to generate 3D region proposals but still use a LiDAR 2D representation to extract features for fusion; several works project LiDAR points on the camera-plane or RGB camera images on the LiDAR BEV plane to align features, whereas许多许多

    “From the tables we have three observations: (1). Most works propose to fuse LiDAR and camera features extracted from 2D convolution neural networks. To do this, they project LiDAR points on the 2D plane and process the feature maps through 2D convolutions. Only a few works extract LiDAR features by PointNet (e.g. [104], [105], [128]) or 3D convolutions (e.g. [123]); (2). Several works on multi”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [104] [104] D. Xu, D. Anguelov, and A. Jain, 'PointFusion: Deep sensor fusion for 3D bounding box estimation,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source

      Not checked in the free run.

    • cites [105] [105] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum PointNets for 3d object detection from RGB-D data,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source

      Not checked in the free run.

    • cites [128] [128] F. Yang, J. Yang, Z. Jin, and H. Wang, 'A fusion model for road detection based on deep learning and fully connected CRF,' in 13th Annu. Conf. System of Systems Engineering . IEEE, 2018, pp. 29-36. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § A. What to Fuse · p. 6

    LiDARs and cameras are the most common sensors for multi-modal perception in the literature.

    “LiDARs and cameras (visual cameras, thermal cameras) are the most common sensors for multi-modal perception in the literature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § A. Deep Multi-modal Object Detection · p. 5

    Some works focus on fusing RGB images from visual cameras with images from thermal cameras.

    “Some other works focus on fusing the RGB images from visual cameras with images from thermal cameras [91], [117]-[119].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [91] [91] K. Takumi, K. Watanabe, Q. Ha, A. Tejero-De-Pablos, Y. Ushiku, and T. Harada, 'Multispectral object detection for autonomous vehicles,' in Proc. Thematic Workshops of ACM Multimedia , 2017, pp. 35-43. -

      Not checked in the free run.

    • cites [117]-[119]

      Not checked in the free run.

  • May combine two claims.

    Findingnone given
    § A. What to Fuse · p. 6

    There are mainly three ways to process LiDAR point clouds: discretizing into 3D voxels, directly learning over 3D points in continuous vector space, and projecting onto 2D grid-based feature maps.

    “There are mainly three ways to process point clouds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § VII. CONCLUSION AND DISCUSSION · p. 14

    There is a lack of research on multi-modal perception in open-set conditions or with sensor failures.

    “Additionally, there is a lack of research on multi-modal perception in openset conditions or with sensor failures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § I. INTRODUCTION · p. 1

    A deep neural network is a powerful tool for learning hierarchical feature representations given a large amount of data.

    “A deep neural network is a powerful tool for learning hierarchical feature representations given a large amount of data [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] [5] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, p. 436, 2015. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges · p. 1

    There is no general guideline for network architecture design for deep multi-modal perception, and questions of what to fuse, when to fuse, and how to fuse remain open.

    “However, there is no general guideline for network architecture design, and questions of 'what to fuse', 'when to fuse', and 'how to fuse' remain open.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § B. Contributions · p. 2

    There is no survey that focuses on deep multi-modal object detection (2D or 3D) and semantic segmentation for autonomous driving.

    “To the best of our knowledge, there is no survey that focuses on deep multi-modal object detection (2D or 3D) and semantic segmentation for autonomous driving, which makes it difficult for beginners to enter this research field.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § C. When to Fuse · p. 9

    It is not easy to find the 'optimal' way to fuse intermediate layers given a specific network architecture.

    “Although the middle fusion approach is highly flexible, it is not easy to find the 'optimal' way to fuse intermediate layers given a specific network architecture.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § C. When to Fuse · p. 8

    Middle fusion combines feature representations from different sensing modalities at intermediate layers.

    “Middle fusion is the compromise of early and late fusion: It combines the feature representations from different sensing modalities at intermediate layers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A. What to Fuse · p. 6

    A camera-plane map (CPM) can be directly fused with camera images because their sizes are the same, but it leaves many pixels empty.

    “A CPM can be produced by projecting the 3D points into the camera coordinate system, provided the calibration matrix. A CPM can be directly fused with camera images, as their sizes are the same. However, this representation leaves many pixels empty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § C. When to Fuse · p. 8

    Middle fusion enables the network to learn cross modalities with different feature representations and at different depths.

    “This enables the network to learn cross modalities with different feature representations and at different depths.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A. Deep Multi-modal Object Detection · p. 5

    3D detection is more challenging than 2D detection because the object's distance to the ego-vehicle needs to be estimated, and therefore accurate depth information provided by LiDAR sensors is highly beneficial.

    “Compared to 2D detection, 3D detection is more challenging since the object's distance to the ego-vehicle needs to be estimated. Therefore, accurate depth information provided by LiDAR sensors is highly beneficial.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § C. When to Fuse · p. 8

    Deep neural networks represent features hierarchically and offer a wide range of choices to combine sensing modalities at early, middle, or late stages.

    “Deep neural networks represent features hierarchically and offer a wide range of choices to combine sensing modalities at early, middle, or late stages (Fig. 8).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § C. When to Fuse · p. 9

    Based on the papers reviewed, feature concatenation is the most common operation, especially at early and middle stages.

    “Based on the papers that we have reviewed, feature concatenation is the most common operation, especially at early and middle stages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § A. Multi-modal Data Preparation · p. 11

    Using large multi-modal datasets with diverse driving conditions, object labels, and sensors can significantly improve the network's accuracy and robustness against changing environments.

    “Therefore, using large multi-modal datasets with diverse driving conditions, object labels, and sensors can significantly improve the network's accuracy and robustness against changing environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given
    § VI. CHALLENGES AND OPEN QUESTIONS · p. 9

    A reliable perception system is the prerequisite for a driverless car to run safely in uncontrolled and complex driving environments.

    “A reliable perception system is the prerequisite for a driverless car to run safely in uncontrolled and complex driving environments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § C. When to Fuse · p. 8

    Early fusion has low computation requirements and a low memory budget.

    “Second, early fusion has low computation requirements and a low memory budget as it jointly processes the multiple sensing modalities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning
    § VI. CHALLENGES AND OPEN QUESTIONS · p. 9

    Developing deep multi-modal perception systems is especially challenging for autonomous driving because it has high requirements in accuracy, robustness, and real-time performance.

    “As discussed in the Introduction (cf. Sec. I), developing deep multi-modal perception systems is especially challenging for autonomous driving because it has high requirements in accuracy, robustness, and real-time performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § VI. CHALLENGES AND OPEN QUESTIONS · p. 9

    The predictions from object detection or semantic segmentation are usually transferred to other modules such as maneuver prediction and decision making.

    “The predictions from object detection or semantic segmentation are usually transferred to other modules such as maneuver prediction and decision making.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § C. When to Fuse · p. 8

    Early fusion learns the joint features of multiple modalities at an early stage, fully exploiting the information of the raw data.

    “First, the network learns the joint features of multiple modalities at an early stage, fully exploiting the information of the raw data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § C. When to Fuse · p. 9

    Camera images have been well investigated with larger labeled dataset and better 2D detection performance compared to object detections from LiDAR point clouds.

    “Compared to object detections from LiDAR point clouds, camera images have been well investigated with larger labeled dataset and better 2D detection performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation
    § A. What to Fuse · p. 6

    A spherical map represents each 3D point densely and compactly, making it suitable for point cloud segmentation, but its size can differ from camera images, making early fusion difficult.

    “A spherical map is obtained by projecting each 3D point onto a sphere, characterized by azimuth and zenith angles. It has the advantage of representing each 3D point in a dense and compact way, making it a suitable representation for point cloud segmentation (e.g. [51]). However, the size of the representation can be different from camera images. Therefore, it is difficult to fuse them at an early”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [51] [51] B. Wu, A. Wan, X. Yue, and K. Keutzer, 'SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3d lidar point cloud,' in IEEE Int. Conf. Robotics and Automation , May 2018, pp. 1887-1893. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § C. When to Fuse · p. 8

    Compared to semantic segmentation, there exist more diverse network architectures and more fusion variants in object detection.

    “Compared to the semantic segmentation where multimodal features are fused at different stages in FCN, there exist more diverse network architectures and more fusion variants in object detection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § C. When to Fuse · p. 8

    Late fusion suffers from high computation cost and memory requirements.

    “However, it suffers from high computation cost and memory requirements.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § A. What to Fuse · p. 7

    Camera images provide rich texture information, but objects can be occluded and scale can vary significantly; for 3D environment inference, bird's eye view might be a better representation.

    “Camera images provide rich texture information of the driving surroundings. However, objects can be occluded and the scale of a single object can vary significantly in the camera image plane. For 3D environment inference, the bird's eye view that is commonly used for LiDAR point clouds might be a better representation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § C. When to Fuse · p. 8

    Late fusion discards rich intermediate features which may be highly beneficial when being fused.

    “In addition, it discards rich intermediate features which may be highly beneficial when being fused.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A. What to Fuse · p. 6

    A bird's eye view (BEV) map avoids occlusion problems, preserves objects' length and width, and directly provides objects' positions on the ground plane, making localization easier; therefore it is widely applied to 3D environment perception.

    “Compared to the above-mentioned feature maps which encode LiDAR information in the front-view, a BEV map avoids occlusion problems because objects occupy different space in the map. In addition, the BEV preserves the objects' length and width, and directly provides the objects' positions on the ground plane, making the localization task easier. Therefore, the BEV map is widely applied to 3D”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § A. Multi-modal Data Preparation · p. 11

    Training a deep neural network on a complex task requires a huge amount of data.

    “Training a deep neural network on a complex task requires a huge amount of data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A. What to Fuse · p. 7

    The networks implicitly learn to align features of different viewpoints, and therefore a well-calibrated sensor setup with accurate spatial and temporal alignment is the prerequisite for accurate multi-modal perception.

    “This indicates that the networks implicitly learn to align features of different viewpoints. Therefore, a well-calibrated sensor setup with accurate spatial and temporal alignment is the prerequisite for accurate multi-modal perception, as will be discussed in Sec. VI-A2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § C. When to Fuse · p. 8

    Late fusion has high flexibility and modularity.

    “Late fusion has high flexibility and modularity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § VII. CONCLUSION AND DISCUSSION · p. 14

    There are various ways to fuse sensing modalities in neural networks, encompassing different sensor representations, fusion operations, and fusion stages.

    “There are various ways to fuse sensing modalities in neural networks, encompassing different sensor representations, cf. Sec. V-A, fusion operations cf. Sec. V-B, and fusion stages, cf. Sec. V-C.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The MPO dataset contains 650 dense scans and 34200 sparse scans.

    “650 scans (dense), 34200 scans (sparse)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KAIST multispectral pedestrian dataset contains 95,328 frames and 103,128 objects.

    “95 , 328 frames, 103 , 128 objects”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § A. Multi-modal Data Preparation · p. 12

    Well-calibrated sensors are the prerequisite for accurate and robust multi-modal perception systems.

    “Well-calibrated sensors are the prerequisite for accurate and robust multi-modal perception systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KAIST multispectral pedestrian dataset uses visual and thermal camera.

    “Visual and thermal camera”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KAIST multispectral pedestrian dataset was collected in 2015.

    “2015”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § B. Fusion Methodology · p. 13

    There is no conclusive evidence that one fusion scheme is better than the others.

    “There is no conclusive evidence that one fusion scheme is better than the others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § B. Fusion Methodology · p. 12

    So far, we are not aware of any work fusing Ultrasonic sensor signals in deep multi-modal perception.

    “So far, we are not aware of any work fusing Ultrasonic sensor signals in deep multi-modal perception, despite its relevance for low-speed scenarios.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § B. Fusion Methodology · p. 13

    So far, BNNs have not been introduced to multi-modal perception.

    “Although much progress has been made for BNNs, to the best of our knowledge, so far they have not been introduced to multi-modal perception.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KITTI dataset includes Car, Van, Truck, Pedestrian, Person (sitting), Cyclist, Tram, Misc.

    “Car, Van, Truck, Pedestrian, Person (sitting), Cyclist, Tram, Misc”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KAIST multispectral pedestrian dataset was collected in Seoul.

    “Seoul”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § B. Fusion Methodology · p. 12

    There are very few studies for fusing LiDAR and camera outputs with signals from other sources such as Radars, ultrasonics or V2X communication.

    “In addition, there are very few studies for fusing LiDAR and camera outputs with signals from other sources such as Radars, ultrasonics or V2X communication.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KITTI dataset contains 7481 training frames and 80,256 objects.

    “7481 frames (training) 80 . 256 objects”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The MPO dataset uses visual camera, LiDAR, and GNSS.

    “Visual camera, LiDAR, and GNSS”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The MPO dataset has no dynamic objects.

    “No dynamic objects”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KITTI dataset uses visual (stereo) camera, 3D LiDAR, GNSS, and inertial sensors.

    “Visual (Stereo) camera, 3D LiDAR, GNSS, and inertial sensors”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The MPO dataset was collected in Fukuoka.

    “Fukuoka”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KAIST multispectral pedestrian dataset is for 2D bounding box.

    “2D bounding box”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The Malaga Stereo and Laser Urban dataset was collected in 2014.

    “2014”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KITTI dataset was collected in 2012, 2013, 2015.

    “2012, 2013, 2015”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KAIST multispectral pedestrian dataset includes Person, People, Cyclist during day and night.

    “Person, People, Cyclist during day and night”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] [90] S. Hwang, J. Park, N. Kim, Y. Choi, and I. So Kweon, 'Multispectral pedestrian detection: Benchmark dataset and baseline,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 1037-1045. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § C. Others · p. 14

    The predictions in a frame are not dependent on previous frames, resulting in inconsistency over time.

    “The predictions in a frame are not dependent on previous frames, resulting in inconsistency over time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KITTI dataset was collected in Karlsruhe.

    “Karlsruhe”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The MPO dataset is for place categorization.

    “Place categorization”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The Malaga Stereo and Laser Urban dataset uses visual (stereo) camera, 5×2D LiDAR (yielding 3D information), GNSS and inertial sensors.

    “Visual (Stereo) camera, 5 × 2D LiDAR (yielding 3D information), GNSS and inertial sensors”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § B. Fusion Methodology · p. 13

    Each method uses different hardware, and the inference time is reported only by the authors.

    “Each method uses different hardware, and the inference time is reported only by the authors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The Malaga Stereo and Laser Urban dataset has no task.

    “no”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § C. Others · p. 14

    These metrics only summarize the prediction accuracy to a test dataset. They do not consider how sensor behaves in different situations.

    “However, these metrics only summarize the prediction accuracy to a test dataset. They do not consider how sensor behaves in different situations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning
    § B. Fusion Methodology · p. 12

    Explicitly modeling uncertainty or informativeness of each sensing modality is important for safe autonomous driving.

    “Explicitly modeling uncertainty or informativeness of each sensing modality is important safe autonomous driving.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The KITTI dataset is for 2D/3D bounding box, visual odometry, road, optical flow, tracking, depth, 2D instance and pixel-level segmentation.

    “2D/3D bounding box, visual odometry, road, optical flow, tracking, depth, 2D instance and pixel-level segmentation”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] [6] A. Geiger, P. Lenz, and R. Urtasun, 'Are we ready for autonomous driving? the KITTI vision benchmark suite,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2012. - source

      Not checked in the free run.

    • cites [75] [75] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, 'Vision meets robotics: The KITTI dataset,' Int. J. Robotics Research , 2013. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The MPO dataset was collected in 2016.

    “2016”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] [77] H. Jung, Y. Oto, O. M. Mozos, Y. Iwashita, and R. Kurazume, 'Multimodal panoramic 3d outdoor datasets for place categorization,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , 2016, pp. 45454550. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The Malaga Stereo and Laser Urban dataset was collected in Malaga.

    “M´ alaga”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § C. Others · p. 14

    Current methods are designed to propagate results to other modules in autonomous driving, such as localization, planning, and reasoning.

    “Furthermore, current methods are designed to propagate results to other modules in autonomous driving, such as localization, planning, and reasoning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A. Multi-modal Data Preparation · p. 12

    Temporal and spatial sensing misalignments could cause severe errors in training datasets and degrade the performance of networks.

    “This could cause severe errors in training datasets and degrade the performance of networks, especially for those which are designed to implicitly learn the sensor alignment (e.g. networks that fuse LiDAR BEV feature maps and front view camera images cf. Sec. V-A3).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The Malaga Stereo and Laser Urban dataset has n.a. objects.

    “n.a.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Qi et al. (2017) [105] achieve Pedestrian detection performance on KITTI test set of 51.21% Easy, 44.89% Moderate, and 40.23% Hard.

    “Qi et al. , 2017 [105] | 70.39 % | 81.20 % | 62.19 % | 44.89 % | 51.21 % | 40.23 % | 56.77 % | 71.96 % | 50.39 % | 0.17 s | GPU @ 3.0 Ghz (Python) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [105] [105] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum PointNets for 3d object detection from RGB-D data,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Liang et al. (2018) [150] achieve Car detection performance on KITTI test set of 82.54% Easy, 66.22% Moderate, and 64.04% Hard.

    “Liang et al. , 2018 [150] | 66.22 % | 82.54 % | 64.04 % | - | - | - | - | - | - | 0.06 s | GPU @ 2.5 Ghz (Python) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [150] [150] M. Liang, B. Yang, S. Wang, and R. Urtasun, 'Deep continuous fusion for multi-sensor 3d object detection,' in Proc. Eur. Conf. Computer Vision , 2018, pp. 641-656. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Sindagi et al. (2019) [113] achieve Car detection performance on KITTI test set of 83.2% Easy, 72.7% Moderate, and 65.12% Hard.

    “Sindagi et al. , 2019 [113] | 72.7 % | 83.2 % | 65.12 % | - | - | - | - | - | - | - | - |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [113] [113] V. A. Sindagi, Y. Zhou, and O. Tuzel, 'MVX-Net: Multimodal voxelnet for 3D object detection,' in IEEE Int. Conf. Robotics and Automation , 2019. - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Ku et al. (2017) [103] achieve Pedestrian detection performance on KITTI test set of 50.80% Easy, 42.81% Moderate, and 40.88% Hard.

    “Ku et al. , 2017 [103] | 71.88 % | 81.94 % | 66.38 % | 42.81 % | 50.80 % | 40.88 % | 52.18 % | 64.00 % | 46.61 % | 0.1 s | GPU Titan X (Pascal) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [103] [103] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander, 'Joint 3d proposal generation and object detection from view aggregation,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , Oct. 2018, pp. 1-8. - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Liang et al. (2019) [116] achieve Cyclist detection performance on KITTI test set of 79.58% Easy, 64.68% Moderate, and 57.03% Hard.

    “Liang et al. , 2019 [116] | 76.75 % | 86.81 % | 68.41 % | 45.61 % | 52.37 % | 41.49 % | 64.68 % | 79.58 % | 57.03 % | 0.08 s | GPU @ 2.5 Ghz (Python) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [116] [116] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun, 'Multi-task multisensor fusion for 3D object detection,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2019, pp. 7345-7353. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Qi et al. (2017) [105] achieve Cyclist detection performance on KITTI test set of 71.96% Easy, 56.77% Moderate, and 50.39% Hard.

    “Qi et al. , 2017 [105] | 70.39 % | 81.20 % | 62.19 % | 44.89 % | 51.21 % | 40.23 % | 56.77 % | 71.96 % | 50.39 % | 0.17 s | GPU @ 3.0 Ghz (Python) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [105] [105] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum PointNets for 3d object detection from RGB-D data,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2018. - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Shin et al. (2018) [141] achieve Car detection performance on KITTI test set of 83.71% Easy, 73.04% Moderate, and 59.16% Hard.

    “Shin et al. , 2018 [141] | 73.04 % | 83.71 % | 59.16 % | - | - | - | - | - | - | - | GPU Titan X (not Pascal) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [141] [141] K. Shin, Y. P. Kwon, and M. Tomizuka, 'RoarNet: A robust 3D object detection based on region approximation refinement,' in IEEE Intelligent Vehicles Symp. , 2018. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE II: OVERVIEW OF MULTI-MODAL DATASETS · p. 22

    The Malaga Stereo and Laser Urban dataset contains 113,082 frames, 5,654.6s (camera); >220,000 frames, 5,000s (LiDARs).

    “113 , 082frames, 5 , 654 . 6s (camera); > 220 , 000frames, 5 , 000s (LiDARs)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] [76] J.-L. Blanco-Claraco, F.- ´ A. Moreno-Due˜ nas, and J. Gonz´ alez-Jim´ enez, 'The M´ alaga urban dataset: High-rate stereo and lidar in a realistic urban scenario,' Int. J. Robotics Research , vol. 33, no. 2, pp. 207-214, 2014. - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Liang et al. (2019) [116] achieve Pedestrian detection performance on KITTI test set of 52.37% Easy, 45.61% Moderate, and 41.49% Hard.

    “Liang et al. , 2019 [116] | 76.75 % | 86.81 % | 68.41 % | 45.61 % | 52.37 % | 41.49 % | 64.68 % | 79.58 % | 57.03 % | 0.08 s | GPU @ 2.5 Ghz (Python) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [116] [116] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun, 'Multi-task multisensor fusion for 3D object detection,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2019, pp. 7345-7353. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Ku et al. (2017) [103] achieve Cyclist detection performance on KITTI test set of 64.00% Easy, 52.18% Moderate, and 46.61% Hard.

    “Ku et al. , 2017 [103] | 71.88 % | 81.94 % | 66.38 % | 42.81 % | 50.80 % | 40.88 % | 52.18 % | 64.00 % | 46.61 % | 0.1 s | GPU Titan X (Pascal) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [103] [103] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander, 'Joint 3d proposal generation and object detection from view aggregation,' in IEEE/RSJ Int. Conf. Intelligent Robots and Systems , Oct. 2018, pp. 1-8. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § TABLE III: SUMMARY OF MULTI-MODAL OBJECT DECTECTION METHODS · p. 27

    Chen et al. (2017) [98] achieve Car detection performance on KITTI test set of 71.09% Easy, 62.35% Moderate, and 55.12% Hard.

    “Chen et al. , 2017 [98] | 62.35 % | 71.09 % | 55.12 % | - | - | - | - | - | - | 0.36 s | GPU @ 2.5 Ghz (Python + C/C++) |”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [98] [98] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, 'Multi-view 3D object detection network for autonomous driving,' in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2017, pp. 6526-6534. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

85 of 85

  • The paper relies on this without support.

    Recent advancements in perception for autonomous driving are driven by deep learning.

    “Recent advancements in perception for autonomous driving are driven by deep learning.”

  • The paper relies on this without support.

    Multiple sensing modalities can be fused to exploit their complementary properties.

    “In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties.”

  • The paper relies on this without support.

    Autonomous driving requires high accuracy, robustness, and real-time performance.

    “developing deep multi-modal perception systems is especially challenging for autonomous driving because it has high requirements in accuracy, robustness, and real-time performance.”

  • The paper relies on this without support.

    A reliable perception system is prerequisite for safe driving.

    “A reliable perception system is the prerequisite for a driverless car to run safely in uncontrolled and complex driving environments.”

  • The paper relies on this without support.

    Training a deep neural network on a complex task requires a huge amount of data.

    “Training a deep neural network on a complex task requires a huge amount of data.”

  • The paper relies on this without support.

    Data quality significantly affects the performance of a deep multi-modal perception system.

    “Besides data diversity and the size of the training dataset, data quality significantly affects the performance of a deep multi-modal perception system as well.”

  • The paper relies on this without support.

    Well-calibrated sensors are prerequisite for accurate and robust multi-modal perception systems.

    “Well-calibrated sensors are the prerequisite for accurate and robust multi-modal perception systems.”

  • The paper relies on this without support.

    Explicitly modeling uncertainty or informativeness of each sensing modality is important for safe autonomous driving.

    “Explicitly modeling uncertainty or informativeness of each sensing modality is important safe autonomous driving.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Other datasets focus exclusively on pedestrian annotations on images.

    “Other datasets focus exclusively on pedestrian annotations on images [20, 25, 79, 24, 88, 23, 58].”

    From nuScenes: A Multimodal Dataset for Autonomous Driving (2020) · cites this paper as [25]

    The Labels section (p. 5) describes a dataset that labels only people and says EuroCity focuses on vulnerable road users, mostly pedestrians; the Sensing Modalities section (p. 4) says the reviewed datasets include camera images. But “exclusively pedestrian” is too narrow: the former also labels cyclists, and EuroCity includes other road users (Table II, p. 21). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Early concatenation can encode the global multimodal context well.

    “early concatenation that can encode the global multimodal context well [188]”

    From Multimodal learning with transformers: A survey (2023) · cites this paper as [188]

  • Two-stream cross-attention can learn cross-modal interaction, but there is no self-attention to the self-context inside each modality.

    “As discussed in [188], two-stream crossattention can learn cross-modal interaction, whereas there is no self-attention to the self-context inside each modality.”

    From Multimodal learning with transformers: A survey (2023) · cites this paper as [188]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reference [123] is classified as both object detection and terrain segmentation

    Minor

    The object-detection section lists [123] among works on 2D object detection, whereas Table IV describes it as 2D off-road terrain segmentation.

  • Dataset size range excludes the 500-frame Astyx dataset

    Minor

    The stated minimum is 1,569 frames, but Table II includes the Astyx HiRes2019 dataset with 500 frames.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. mackowiak, 2018, CEREALS–cost-effective region-based active learning for semantic segmentation, Proc Brit Mach Vis Conf, 1
  2. sermanet, 2013, OverFeat: Integrated recognition, localization and detection using convolutional networks, Proc Int Conf Learn Representations, 1
  3. lee, 2019, Spigan: Privileged adversarial learning from simulation, Proc Int Conf Learn Representations, 1
  4. roddick, 2019, Orthographic feature transform for monocular 3D object detection, Proc Brit Mach Vis Conf, 1
  5. berger, 2014, Autonomous Driving
  6. dai, 2016, R-FCN: Object detection via region-based fully convolutional networks, Proc Adv Neural Inf Process Syst, 379
  7. 2019, PandaSet Public Large-Scale Dataset for Autonomous Driving
  8. 2019, Waymo open dataset An autonomous driving dataset
  9. kesten, 2019, Lyft level 5 av dataset 2019
  10. geyer, 2019, A2D2 AEV Autonomous Driving Dataset
  11. ramachandram, 2017, Structure optimization for deep multimodal fusion networks using graph-induced kernels, Proc 25th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
  12. valada, 2019, Self-supervised model adaptation for multimodal semantic segmentation, Int J Comput Vis, 127, 1
  13. enderich, 2019, Learning multimodal fixed-point weights using gradient descent, Proc 27th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
  14. qi, 2017, PointNet++: Deep hierarchical feature learning on point sets in a metric space, Proc Adv Neural Inf Process Syst, 5099
  15. li, 2018, PointCNN: Convolution on $\chi$ -transformed points, Proc Adv Neural Inf Process Syst, 826
  16. sauer, 2018, Conditional affordance learning for driving in urban environments, Proc 2st Annu Conf Robot Learn, 237
  17. 2017, On the Road to Fully Self-Driving Waymo Safety Report

Cited by

Sorted by citations

Related

Authors
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K.
Affiliations
  • Universität Ulm (DE)
  • Karlsruhe Institute of Technology (DE)
  • Robert Bosch (Germany) (DE)
ORCID
Venue
IEEE Transactions on Intelligent Transportation Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021-03
Volume
22
Issue
3
Pages
1341-1360
ISSN
  • 1524-9050 (ISSN-L)
  • 1558-0016
DOI
10.1109/tits.2020.2972974
OpenAlex
W3008115128
Cited by
1,425
References
266
Citations by year
2026: 178 · 2025: 274 · 2024: 308 · 2023: 263 · 2022: 181 · 2021: 144 · 2020: 67 · 2019: 10
FWCI
60.51
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations
  • IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Robert Bosch GmbH
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multisensory perception, autonomous driving, object detection, semantic segmentation, sensor fusion, LiDAR, radar, camera, fusion method, onboard sensors, benchmark dataset, scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review