The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks
Open accessAbstract
Scene understanding plays a crucial role in autonomous driving by utilizing sensory data for contextual information extraction and decision making. Beyond modeling advances, the enabler for vehicles to become aware of their surroundings is the availability of visual sensory data, which expand the vehicular perception and realizes vehicular contextual awareness in real-world environments. Research directions for scene understanding pursued by related studies include person/vehicle detection and segmentation, their transition analysis, lane change, and turns detection, among many others. Unfortunately, these tasks seem insufficient to completely develop fully-autonomous vehicles i.e., achieving level-5 autonomy, travelling just like human-controlled cars. This latter statement is among the conclusions drawn from this review paper: scene understanding for autonomous driving cars using vision sensors still requires significant improvements. With this motivation, this survey defines, analyzes, and reviews the current achievements of the scene understanding research area that mostly rely on computationally complex deep learning models. Furthermore, it covers the generic scene understanding pipeline, investigates the performance reported by the state-of-the-art, informs about the time complexity analysis of avant garde modeling choices, and highlights major triumphs and noted limitations encountered by current research efforts. The survey also includes a comprehensive discussion on the available datasets, and the challenges that, even if lately confronted by researchers, still remain open to date. Finally, our work outlines future research directions to welcome researchers and practitioners to this exciting domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Self-citationVision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00075 · Cited by 6,612
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · cited 1×
“…formulations of deep neural networks [123], simpler mechanisms to approximate the output confidence of neural networks (e.g. Monte Carlo dropout [124] or ensembles [125]) and other assorted methods for uncertainty quantification [126] should be progressively incorporated as an additional yet…”
BackgroundSqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 2×
“Lately the computational complexity has been reduced significantly by some deep models such as SqueezeNet [25], which achieves AlexNet level accuracy with 50 times less number of parameters.”
MethodSimple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111 · cited 1×
“Monte Carlo dropout [124] or ensembles [125]) and other assorted methods for uncertainty quantification [126] should be progressively incorporated as an additional yet crucial criterion for decision making.”
BackgroundWhat Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., & Darrell, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00271 · Cited by 2,641
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · cited 2×
“For instance, Ranftl et al. [162] establish an unprecedented state-of-the-art level of performance by introducing vision transformers in a semantic segmentation domain.”
BackgroundSemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., & Gall, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00939 · Cited by 2,294
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826 · cited 3×
“More recently, semantic segmentation achieved significant mile-stones from the perspective of time complexity, as reported in [25] and [26].”
BackgroundMethodICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709
DenseASPP for Semantic Segmentation in Street Scenes
Yang, M., Yu, K., Zhang, C., Li, Z., & Yang, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00388 · Cited by 1,674
The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes
Neuhold, G., Ollmann, T., Bulo, S. R., & Kontschieder, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.534 · Cited by 1,434
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169
Simultaneous Localization and Mapping: A Survey of Current Trends in Autonomous Driving
Bresson, G., Alsayed, Z., Yu, L., & Glaser, S. (2017) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2017.2749181 · Cited by 885
The ApolloScape Dataset for Autonomous Driving
Huang, X., Cheng, X., Geng, Q., Cao, B., Zhou, D., Wang, P., Lin, Y., & Yang, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00141 · Cited by 679
The ApolloScape Open Dataset for Autonomous Driving and its Application
Huang, X., Wang, P., Cheng, X., Zhou, D., Geng, Q., & Yang, R. (2018) · arXiv · doi:10.48550/arxiv.1803.06184 · Cited by 615
The ApolloScape Open Dataset for Autonomous Driving and Its Application
Huang, X., Wang, P., Cheng, X., Zhou, D., Geng, Q., & Yang, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2926463 · Cited by 615
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding
Sakaridis, C., Dai, D., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01059 · Cited by 572
A Comparative Analysis of LiDAR SLAM-Based Indoor Navigation for Autonomous Vehicles
Zou, Q., Sun, Q., Chen, L., Nie, B., & Li, Q. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3063477 · Cited by 321
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01597 · Cited by 216
Applications and Challenges in Video Surveillance via Drone: A Brief Survey
Dilshad, N., Hwang, J., Song, J., & Sung, N. (2020) · 2020 International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc49870.2020.9289536 · Cited by 139 · cited 1×
“In real-world environments [103], devices are severely resource-restricted [104], such as Raspberry-pi, Jetson Nano, and Google Board.”
BackgroundAre We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey of Datasets and Methods
Gao, B., Pan, Y., Li, C., Geng, S., & Zhao, H. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3076844 · Cited by 105
FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data
Krispel, G., Opitz, M., Waltner, G., Possegger, H., & Bischof, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093584 · Cited by 99 · cited 1×
“For instance, point clouds [132], [133], meshes [134] and depth data [135] together with RGB data can generate an increased 3D scene understanding for an autonomous vehicle [136].”
BackgroundMODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving
Siam, M., Mahgoub, H., Zahran, M., Yogamani, S., Jagersand, M., & El-Sallab, A. (2018) · 2018 21st International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569744 · Cited by 95
Traffic Jam Probability Estimation Based on Blockchain and Deep Neural Networks
Hassija, V., Gupta, V., Garg, S., & Chamola, V. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2988040 · Cited by 77
Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime
Dai, D., & Van Gool, L. (2018) · ETH Zurich · doi:10.3929/ethz-b-000310974 · Cited by 60
Visual Scene Understanding for Autonomous Driving Using Semantic Segmentation
Hofmarcher, M., Unterthiner, T., Arjona-Medina, J., Klambauer, G., Hochreiter, S., & Nessler, B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_15 · Cited by 51
SPEED: A Deep Learning Assisted Privacy-Preserved Framework for Intelligent Transportation Systems
Usman, M., Jan, M. A., & Jolfaei, A. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3031721 · Cited by 32 · cited 1×
“Accumulating these information provide more enhanced and safer instructions for automated actions of the vehicle, such as turning manoeuvres, lane changing, or braking [4].”
Background
Cited by
YOLOdrive: A Lightweight Autonomous Driving Single-Stage Target Detection Approach
Wang, L., Hua, S., Zhang, C., Yang, G., Ren, J., & Li, J. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3439863 · Cited by 14
Metadata
- Authors
- Muhammad, K., Hussain, T., Ullah, H., Ser, J. D., Rezaei, M., Kumar, N., Hijji, M., Bellavista, P., & de Albuquerque, V. H. C.
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2022-12
- Volume
- 23
- Issue
- 12
- Pages
- 22694-22715
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W4311166797
- Cited by
- 210
- References
- 39
- FWCI
- 13.60
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.