Sensing and Machine Learning for Automotive Perception: A Review
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Automotive perception involves understanding the external driving environment and the internal state of the vehicle cabin and occupants using sensor data. It is critical to achieving high levels of safety and autonomy in driving. This article provides an overview of different sensor modalities, such as cameras, radars, and light detection and ranging (LiDAR) used commonly for perception, along with the associated data processing techniques. Critical aspects of perception are considered, such as architectures for processing data from single or multiple sensor modalities, sensor data processing algorithms and the role of machine learning techniques, methodologies for validating the performance of perception systems, and safety. The technical challenges for each aspect are analyzed, emphasizing machine learning approaches, given their potential impact on improving perception. Finally, future research opportunities in automotive perception for their wider deployment are outlined.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 1×
“Datasets Year Number of images Viewing Angle Resolution Camera Type Adverse Weather CamVid [90] 2008 18× 103 Dashboard 960× 720 Monocular No KITTI [91] 2012 15× 103 Vehicle Roof 1392× 512 2 Stereo No Cityscapes [92] 2016 25× 103 Windshield – Stereo No”
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · cited 1×
“This leads to Bayesian approaches, such as dropout at runtime [185] or ensemble learning.”
MethodObject Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545 · cited 3×
“Directly mapping from image pixels to bounding boxes with category probabilities generally saves time compared with the two-stage methods [103].”
BackgroundMethodInfluentialCenterNet: Keypoint Triplets for Object Detection
Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., & Tian, Q. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00667 · Cited by 4,079
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · cited 2×
“To overcome issues with real-world testing, synthetic data can be obtained in virtual environments [100], [164], [165], [166], e.g., using photorealistic simulation or emulating humans and motion behaviors.”
BackgroundObject Detection in 20 Years: A Survey
Zou, Z., Chen, K., Shi, Z., Guo, Y., & Ye, J. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3238524 · Cited by 3,689 · cited 1×
“…improvement include attention mechanism, pyramid pooling, linear bottleneck and inverted residuals, depthwise separable convolution, atrous convolution, knowledge distillation, domain adaptation, SEBlock, ResBlock, mask mechanism, network pruning, and quantification [110], [118] [119], [120].”
BackgroundSimple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450
Deep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01247-4 · Cited by 2,965 · cited 2×
“Studies [107] and [108] show that CNN has a remarkable ability to locate objects in convolution layers.”
BackgroundDeep Learning for Generic Object Detection: A Survey
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2018) · arXiv · doi:10.48550/arxiv.1809.02165 · Cited by 2,965
Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics
Bernardin, K., & Stiefelhagen, R. (2008) · EURASIP Journal on Image and Video Processing · doi:10.1155/2008/246309 · Cited by 2,918 · cited 1×
“Moreover, tracking is a crucial step for perception, so tracking metrics, such as multiple object tracking accuracy/multiple object tracking precision, are also employed [155].”
BackgroundBDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., & Darrell, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00271 · Cited by 2,641
A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
Lee, K., Lee, K., Lee, H., & Shin, J. (2018) · arXiv · doi:10.48550/arxiv.1807.03888 · Cited by 2,450
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., & Gall, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00939 · Cited by 2,294 · cited 1×
“However, different annotated datasets have recently emerged including the SemanticKITTI dataset [137].”
BackgroundA Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1610.02136 · Cited by 1,968
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,815
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632 · cited 1×
“Oxford RobotCar [93] 2017 19× 103 360 Degree 1280× 960 Stereo & Monocular Yes 1024× 1024 Mapillary [94] 2017 25× 103 – 1920× 1080 – Yes BDD100K [95] 2017 100× 106 Windshield 720p Video Yes ApolloScape [96] 2018 144× 103 360 Degree 3384× 2710 Stereo No CULane [97] 2018 130× 103 Windshield 1640× 590 – No H3D [98] 2019 2…”
The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes
Neuhold, G., Ollmann, T., Bulo, S. R., & Kontschieder, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.534 · Cited by 1,434 · cited 1×
“Oxford RobotCar [93] 2017 19× 103 360 Degree 1280× 960 Stereo & Monocular Yes 1024× 1024 Mapillary [94] 2017 25× 103 – 1920× 1080 – Yes BDD100K [95] 2017 100× 106 Windshield 720p Video Yes ApolloScape [96] 2018 144× 103 360 Degree 3384× 2710 Stereo No CULane [97] 2018 130× 103 Windshield 1640× 590 – No H3D [98] 2019 2…”
RepPoints: Point Set Representation for Object Detection
Yang, Z., Liu, S., Hu, H., Wang, L., & Lin, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00975 · Cited by 1,307 · cited 1×
“Compared to RepPoints [115] using nine keypoints, Yang et al. [116] used a large number of adaptive points to model objects, which achieves state-of-the-art performance on instance segmentation tasks.”
MethodSegmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050 · cited 1×
“Datasets Year Number of images Viewing Angle Resolution Camera Type Adverse Weather CamVid [90] 2008 18× 103 Dashboard 960× 720 Monocular No KITTI [91] 2012 15× 103 Vehicle Roof 1392× 512 2 Stereo No Cityscapes [92] 2016 25× 103 Windshield – Stereo No”
Recent advances in deep learning for object detection
Wu, X., Sahoo, D., & Hoi, S. C. H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.085 · Cited by 1,047
The ApolloScape Open Dataset for Autonomous Driving and its Application
Huang, X., Wang, P., Cheng, X., Zhou, D., Geng, Q., & Yang, R. (2018) · arXiv · doi:10.48550/arxiv.1803.06184 · Cited by 615
Self-citationThe ApolloScape Open Dataset for Autonomous Driving and Its Application
Huang, X., Wang, P., Cheng, X., Zhou, D., Geng, Q., & Yang, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2926463 · Cited by 615
Self-citationTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · cited 1×
“Aspects such as explainability and reproducibility of ML processing become critical in ADAS/ADS and, given their safety-critical nature, remain a challenge [194], [195].”
BackgroundCenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection
Nabati, R., & Qi, H. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00157 · Cited by 461 · cited 2×
“D object detection in comparison to [144] and individual sensors in adverse weather and illumination conditions.”
BackgroundMethodThe Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset
Barnes, D., Gadd, M., Murcutt, P., Newman, P., & Posner, I. (2019) · arXiv · doi:10.48550/arxiv.1909.01300 · Cited by 443
Exploring Categorical Regularization for Domain Adaptive Object Detection
Xu, C.-D., Zhao, X.-R., Jin, X., & Wei, X.-S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01174 · Cited by 355
Zhou, X., Wang, D., & Krähenbühl, P. (2019) · arXiv · doi:10.48550/arxiv.1904.07850 · Cited by 288 · cited 1×
“These selected temporally enhanced features are then used to regress oriented bounding boxes (OBB) at each of successive radar frames, similar to the CenterPoint framework [85].”
MethodThe H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes
Patil, A., Malla, S., Gang, H., & Chen, Y.-T. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793925 · Cited by 263 · cited 1×
“Oxford RobotCar [93] 2017 19× 103 360 Degree 1280× 960 Stereo & Monocular Yes 1024× 1024 Mapillary [94] 2017 25× 103 – 1920× 1080 – Yes BDD100K [95] 2017 100× 106 Windshield 720p Video Yes ApolloScape [96] 2018 144× 103 360 Degree 3384× 2710 Stereo No CULane [97] 2018 130× 103 Windshield 1640× 590 – No H3D [98] 2019 2…”
Multimodal vehicle detection: fusing 3D-LIDAR and color camera data
Asvadi, A., Garrote, L., Premebida, C., Peixoto, P., & J. Nunes, U. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.09.038 · Cited by 225 · cited 1×
“In [142], a multimodal vehicle detection system employing a late fusion strategy was proposed combining optical image and 3-”
MethodPrecise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception
Hurl, B., Czarnecki, K., & Waslander, S. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8813809 · Cited by 122
Federated Deep Learning Meets Autonomous Vehicle Perception: Design and Verification
Wang, S., Li, C., Ng, D. W. K., Eldar, Y. C., Poor, H. V., Hao, Q., & Xu, C. (2023) · IEEE Network · doi:10.1109/mnet.104.2100403 · Cited by 83
Metadata
- Authors
- Pandharipande, A., Cheng, C.-H., Dauwels, J., Gurbuz, S. Z., Ibanez-Guzman, J., Li, G., Piazzoni, A., Wang, P., & Santra, A.
- Venue
- IEEE Sensors Journal
- Published
- 2023-06-01
- Volume
- 23
- Issue
- 11
- Pages
- 11097-11115
- ISSN
- 1530-437X (ISSN-L)
- 1558-1748
- 2379-9153
- OpenAlex
- W4362009298
- Cited by
- 161
- References
- 36
- FWCI
- 11.62
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.